:W&Bá¾(-s&M booerk48ª; 9 /0.29.0+7872fd9d9f86ca5c50617e19cb168122b7f959480.70.0!ØXø‚" cf7ipraa6ivo booerk48ŠÔ booerk48juju11-uc-san-diegoSPAtop"˜ _wandb‚{}  batch_size‚256  dataset_limit‚0.2  dropout‚0.2468656070897737 epochs‚10  hidden_dim‚64 ' learning_rate‚0.0008244527021438164  num_attention_heads‚8  num_branch_encoder_layers‚7  num_embedding_layers‚10  num_encoder_layers‚6Bjjtttt_blocks_booerk48bs2s2ev0zj*jj-spatop-sweepagent-tttt-pairwise-1-8pwgvŠ Ê–çÔ¨Œ^š3 (H¨¸hp"3.12.10*0.29.0j linux-x86_64iüpy7 booerk48&  _runtime‚0  _wandb.runtime‚07»ÉÒÀ€ˆ€’NVIDIA A100-SXM4-80GB˜¢ /€€€øáo€€ýÒú\ª€àäêÞÂS NVIDIA A100-SXM4-80GB€€€€À€6"Ampere*(GPU-c1fdb213-5a16-84e3-13b4-cf29bc35dbf8Ê13.2º m8uxme75dd0xhxvdv5byi0jz14m8vxdtµª½Ç2  requirements.txt¢=AW booerk48ZF (H¨¸ (H¨¸ (hpð"3.12.10*0.29.0j linux-x86_64e¾)ÛY booerk48ZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.29.0j linux-x86_64^K²®*  booerk48b trainer/global_step2°ÿÜõ\  booerk48ZK (H¨¸ (H¨¸(8hpð"3.12.10*0.29.0j linux-x86_64që®/  booerk48b*"trainer/global_step25òY1K  booerk48j: Ê–çÔð‚ë*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] ÄšõÆ’  booerk48j€  Ë–çÔèû_î â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 220 K │ train │ 0 │ │ 1 │ encoder │ JetEncoderWithPairwise │ 498 K │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 13.6 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 48 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ ¯/dT[ booerk48jÉ Ë–çÔ˜¡ó`·Trainable params: 14.3 M Non-trainable params: 86 Total params: 14.3 M Total estimated model params size (MB): 57.161 Modules in train mode: 5928 Modules in eval mode: 0 Total FLOPs: 0 f¾™p booerk48j_ Ë–çԘ߈aPwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). Ó¢å:z booerk48ji Ë–çÔø°aZwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). ÕæÍ~x booerk48jg Ë–çÔø¤‘aXwandb: WARNING Config item 'num_encoder_layers' was locked by 'sweep' (ignored update). ´t™¯ booerk48jn Ë–çÔØù”a_wandb: WARNING Config item 'num_branch_encoder_layers' was locked by 'sweep' (ignored update). Q7#Óy booerk48jh Ë–çÔ辘aYwandb: WARNING Config item 'num_attention_heads' was locked by 'sweep' (ignored update). 'Ïõâs booerk48jb Ë–çÔ¨ñœaSwandb: WARNING Config item 'dataset_limit' was locked by 'sweep' (ignored update). Ç*p booerk48j_ Ë–çԈƠaPwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). y,³˜s booerk48jb Ë–çÔÈû£aSwandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). 6[‡Ÿm booerk48j\ Ë–çÔØÀ§aMwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). Œ¯…žl booerk48j[ Ë–çÔ˜öªaLwandb: WARNING Config item 'epochs' was locked by 'sweep' (ignored update). ö±º>Ù booerk48*Ç  transformer_dim‚64  transformer_dim_scale‚2.0  initial_embedding_dim‚64  position_embedding_dim‚64 ! num_branch_embedding_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true  use_pairwise_interactions‚1  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚"all"  pairwise_block_embeddings‚1  pairwise_cross_type‚true  pairwise_feature_set‚""  pairwise_eq2to2‚false  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 F event_info_file‚2"/data/spatop/event_files/tttt/tttt_hadronic.yaml" = training_file‚+"/data/spatop/tttt_15M/h5/tttt_training.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"{ðh†ê booerk48jØ Ë–çÔˆÿøgÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. §{hÐM booerk48j» Ë–çÔÈŠŠh«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. Ö¬òìë booerk48jÙ ΖçÔطɧÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. -2£P booerk48j¾ ΖçÔàŠå§­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. š¼“õ‚(:† Ù–çÔâ✈ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚104.826' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚26.2065 gpu.0.smClock‚1215 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬ÂÓ½å‚(:Û Ù–çÔ϶­Ž disk.rbd2.out‚0 disk.sdb1.out‚0 disk.sda.in‚0 network.sent‚39928$ disk./.usageGB‚2974.662021636963 disk.md0p1.in‚0 disk.sdb2.in‚0 disk.sdb1.in‚0 disk.nvme0n1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.nvme1n1.in‚0 disk.sda2.out‚ 1.6904296875 disk.md0.out‚ 1.63671875 disk.rbd2.in‚0 disk.sdb.out‚ 1.6904296875 disk.nvme0n1.out‚0.609375 disk.sda2.in‚0 proc.cpu.threads‚17 disk.sdb2.out‚ 1.6904296875 disk.nvme1n1.out‚ 375.7421875 disk.sdb.in‚0 disk.dm-0.in‚0 disk.sda1.in‚0 disk.md0p1.out‚ 1.63671875* proc.memory.percent‚17.890477180480957 cpu‚20.093918968578812 network.recv‚53966( disk./.usagePercent‚83.1727951177464 disk.md0.in‚0 disk.sda.out‚ 1.6904296875 disk.dm-0.out‚ 376.3515625$ memory_percent‚79.06138896942139# proc.memory.rssMB‚ 2931.17578125 disk.rbd0.out‚0) proc.memory.availableMB‚ 3430.582031250Lô÷ ‚(:í è–çÔ­ð‚ˆ disk.sda2.out‚ 3.54833984375" proc.memory.rssMB‚ 2931.6640625 proc.cpu.threads‚17 disk.md0.out‚3.4375 disk.rbd2.in‚0 disk.sda2.in‚0 disk.md0p1.in‚0 disk.md0.in‚0 disk.sdb.in‚0 disk.sdb1.in‚0 disk.dm-0.out‚ 511.1015625 disk.sda1.in‚0* proc.memory.percent‚17.893457412719727$ disk./.usageGB‚2974.131824493408 disk.rbd0.out‚0 disk.sda.out‚ 3.54833984375 cpu‚21.247545889632296 network.sent‚60094 network.recv‚73278) disk./.usagePercent‚83.15797058370786 disk.sdb1.out‚0 disk.sda1.out‚0 disk.rbd2.out‚0 disk.nvme1n1.out‚ 509.8515625 disk.nvme0n1.in‚23.03125 disk.dm-0.in‚ 23.04296875$ memory_percent‚67.15643405914307) proc.memory.availableMB‚ 5381.08984375 disk.rbd0.in‚0 disk.nvme1n1.in‚ 0.01171875 disk.md0p1.out‚3.4375 disk.sdb2.in‚0 disk.sdb.out‚ 3.54833984375 disk.nvme0n1.out‚1.25 disk.sdb2.out‚ 3.54833984375 disk.sda.in‚0…\ !‚(:† è–çÔÈ™ˆ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚91.463' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚22.86575 gpu.0.smClock‚1200 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ü>6/‘"‚(:‡ ÷–çÔ¢¬à‡ gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚102.935' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.73375 gpu.0.smClock‚1245 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 &2#‚(:¨ ÷–çÔ­õɉ disk.md0.in‚ 0.07421875 disk.md0.out‚ 5.07421875% memory_percent‚58.592844009399414) disk./.usagePercent‚83.15796237084689 disk.sdb1.in‚0 disk.dm-0.out‚ 693.87890625 network.recv‚77717 disk.rbd2.out‚0 disk.nvme1n1.in‚ 0.02734375 disk.sdb.in‚ 0.0390625 disk.md0p1.in‚ 0.07421875 disk.rbd0.out‚0 network.sent‚66069# proc.memory.rssMB‚ 2931.66796875 proc.cpu.threads‚17% disk./.usageGB‚2974.1315307617188 disk.sda.out‚ 5.2333984375 disk.dm-0.in‚38.25 disk.sda1.in‚0 disk.sda2.in‚ 0.03515625 disk.sda2.out‚ 5.2333984375 disk.md0p1.out‚ 5.07421875 disk.sdb2.out‚ 5.2333984375 disk.sdb2.in‚ 0.0390625 disk.rbd2.in‚0 disk.sdb.out‚ 5.2333984375 disk.sdb1.out‚0 disk.nvme0n1.in‚38.21875 disk.nvme0n1.out‚ 1.4140625 disk.rbd0.in‚0 cpu‚21.952077701614957( proc.memory.availableMB‚ 6784.1484375* proc.memory.percent‚17.893481254577637! disk.nvme1n1.out‚ 692.46484375 disk.sda.in‚ 0.03515625 disk.sda1.out‚0wÏĤ‘$‚(:‡ †—çÔ™æÒ‡ gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚133.017' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.25425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì&yO;%‚(:± †—çÔ¯¸ÅŠ disk.dm-0.in‚ 77.16015625 disk.sdb1.in‚0) disk./.usagePercent‚83.15874973889503 disk.dm-0.out‚ 909.07421875 disk.sdb2.out‚ 6.86767578125 disk.rbd0.out‚0 disk.sdb.out‚ 6.86767578125! proc.memory.rssMB‚ 2931.671875* proc.memory.percent‚17.893505096435547 network.recv‚108748 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda.out‚ 6.86767578125 disk.md0p1.out‚ 6.65234375 disk.md0.out‚ 6.65234375 disk.sdb1.out‚0 disk.sda.in‚ 0.23828125) proc.memory.availableMB‚ 8160.44140625 disk.sdb2.in‚0.109375 disk.rbd2.in‚0 proc.cpu.threads‚17% disk./.usageGB‚2974.1596908569336 disk.sda1.in‚0 disk.sda2.out‚ 6.86767578125 disk.nvme1n1.in‚ 3.8671875 cpu‚22.433173331300925 network.sent‚86753 disk.nvme0n1.out‚ 82.546875 disk.sda2.in‚ 0.23828125 disk.md0p1.in‚ 0.34765625 disk.md0.in‚ 0.34765625 disk.rbd2.out‚0! disk.nvme1n1.out‚ 826.52734375$ memory_percent‚50.19261837005615 disk.nvme0n1.in‚ 73.29296875 disk.sdb.in‚0.109375V“3&b/ lr-AdamW/pg1"trainer/global_step2:JP;¼$õ3'b/ lr-AdamW/pg2"trainer/global_step2:JPµü¨à-() _wandbruntime‚64  _runtime‚640«‘­¿)º $ _timestamp‚1.788464010989879e+09 _runtime‚ 64.175100471 _step‚0 & lr-AdamW/pg1‚3.341454293221423e-05 & lr-AdamW/pg2‚3.341454293221423e-05 trainer/global_step‚49…”Á*¼ & lr-AdamW/pg1‚3.341454293221423e-05 & lr-AdamW/pg2‚3.341454293221423e-05 trainer/global_step‚49 $ _timestamp‚1.788464010989879e+09 _runtime‚ 64.175100471 _step‚0{dËq+bm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JP]±²_i,be BDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current"trainer/global_step2:JPa*zRr-bn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JPlWcËg.bc @DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak"trainer/global_step2:JP‡´c/b_ l7bh EDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JPβ”˜W8bS 0DeviceStatsMonitor.on_train_batch_start/num_ooms"trainer/global_step2:JP /[Yg9bc @DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed"trainer/global_step2:JP¹bd ADeviceStatsMonitor.on_train_batch_start/oversize_segments.current"trainer/global_step2:JPüÎ&¯r?bn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JP¯´ •m@bi FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JP³@Ý­mAbi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JPÀ&VkkBbg DDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated"trainer/global_step2:JP‰–ˆ¨lCbh EDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JPtâ pDbl IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JPQŠilEbh EDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JPŠìî¶lFbh EDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current"trainer/global_step2:JPKTbökGbg DDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated"trainer/global_step2:JP-o^„hHbd ADeviceStatsMonitor.on_train_batch_start/active.small_pool.current"trainer/global_step2:JP‡6BeIba >DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JPd@iJbe BDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current"trainer/global_step2:JP%à9vKbr ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JPxÚ loLbk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JPñ šEjMbf CDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed"trainer/global_step2:JP ק#mNbi FDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JPc»œÈtObp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPR/H–jPbf CDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current"trainer/global_step2:JP_™ú5aQb] :DeviceStatsMonitor.on_train_batch_start/active.all.current"trainer/global_step2:JPÆqÛ’iRbe BDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current"trainer/global_step2:JP±B»sSbo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JP­UUó`Tb\ 9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries"trainer/global_step2:JPT„½.kUbg DDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current"trainer/global_step2:JPËtHÙqVbm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JPvü¦qnWbj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JPb°m¶iXbe BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JPþÉ`Yb\ 9DeviceStatsMonitor.on_train_batch_start/segment.all.freed"trainer/global_step2:JP ­TgZbc @DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak"trainer/global_step2:JP°’¸p[bl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JPÒ •c\b_ ?v`br ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JPCﲤcab_ DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JP^ànpbj GDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current"trainer/global_step2:JP½o—ânqbj GDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated"trainer/global_step2:JPYl9lrbh EDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current"trainer/global_step2:JPo"Lîpsbl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JPo¤oŒgtbc @DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current"trainer/global_step2:JP”\Ø`mubi FDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JPä}ÏYhvbd ADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak"trainer/global_step2:JP+úÆýswbo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JPý¬%nxbj GDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current"trainer/global_step2:JPrjclybh EDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JPg8Šizbe BDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak"trainer/global_step2:JPøå1r{bn KDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JP‘¹,‘n|bj GDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JP|{Æs}bo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JP¯Q?g~bc @DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated"trainer/global_step2:JP0ð)gbc @DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed"trainer/global_step2:JP¶hãli€bd ADeviceStatsMonitor.on_train_batch_start/active.large_pool.current"trainer/global_step2:JP=)@Èobj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JPb™Ýap‚bk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JP2M0kƒbf CDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed"trainer/global_step2:JPÕ»°ao„bj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JP¤>¤n…bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JPíÎñc†b^ ;DeviceStatsMonitor.on_train_batch_start/segment.all.current"trainer/global_step2:JP  c‡b^ ;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak"trainer/global_step2:JP¾õû™^ˆbY 6DeviceStatsMonitor.on_train_batch_start/max_split_size"trainer/global_step2:JP/6J`‰b[ 8DeviceStatsMonitor.on_train_batch_start/segment.all.peak"trainer/global_step2:JPš¦wZhŠbc @DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed"trainer/global_step2:JP¥²…,g‹bb ?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed"trainer/global_step2:JP³y=ƒgŒbb ?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak"trainer/global_step2:JPP¿ý×eb` =DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak"trainer/global_step2:JP¥Ý}ÿqŽbl IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JPŽ%zÕfba >DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JPÿW€Ãeb` =DeviceStatsMonitor.on_train_batch_start/segment.all.allocated"trainer/global_step2:JP±í>{g‘bb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JP£Nñl’bg DDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated"trainer/global_step2:JP!fŸ…t“bo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP8hêk”bf CDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated"trainer/global_step2:JPKI@Zl•bg DDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JPêÎÕ g–bb ?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak"trainer/global_step2:JPÑNs—bn KDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JPž(Š_˜bZ 7DeviceStatsMonitor.on_train_batch_start/num_device_free"trainer/global_step2:JPôÕUvf™ba >DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JPÚ¡Çošbj GDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated"trainer/global_step2:JP3;jÝg›bb ?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak"trainer/global_step2:JP•€>Jiœbd ADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed"trainer/global_step2:JPàÿË@kbf CDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated"trainer/global_step2:JPÁ³,Zjžbe BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated"trainer/global_step2:JP.}|NrŸbm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JPµŠ!g bb ?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed"trainer/global_step2:JPXH}t¡bo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JPÓW¤k¢bf CDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated"trainer/global_step2:JPuÏ4'f£ba >DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JP(Tlb¤b] :DeviceStatsMonitor.on_train_batch_start/num_oom_rejections"trainer/global_step2:JP>ášYp¥bk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JPS ¥¸.¦) _wandbruntime‚64  _runtime‚64ãB£þ §€O \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 97376611840 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051901 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30506 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚926241 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 _runtime‚ 64.176326747 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 97096430832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 97143710208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30506 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚548820 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚548818 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 1982053658624 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2277587 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 97143710208 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1326500 UGDeviceStatsMo"#=|nitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281221 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _step‚1 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30749 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1326500 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966132736 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 1982053658624 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚899466 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 297953792 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚350646 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚49 % _timestamp‚1.7884640109912453e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1351346 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1284 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 1986974270336 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1351346 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 2079430270464 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 1890123852400 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚926241 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 1890106813040 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966132736 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 2079179365888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚697 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 96867457296 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚926239 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1265 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 2101297963520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273¦09Š'¨„O OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1326500 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281221 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _step‚1 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30749 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1326500 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966132736 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 1982053658624 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚899466 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 297953792 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚350646 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚49 % _timestamp‚1.7884640109912453e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1351346 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1284 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 1986974270336 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1351346 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 2079430270464 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 1890123852400 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚926241 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 1890106813040 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966132736 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 2079179365888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚697 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 96867457296 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚926239 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1265 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 2101297963520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 97376611840 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051901 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30506 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚926241 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 _runtime‚ 64.176326747 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 97096430832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 97143710208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30506 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚548820 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚548818 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 1982053658624 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2277587 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 97143710208¯‡Ÿg©bb ?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed"trainer/global_step2:JP†*Ágfªba >DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JPa~h«bc @DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak"trainer/global_step2:JPy ýNm¬bh EDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JPvÛ–†m­bh EDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JPŒCÕfr®bm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP!äö]¯bX 5DeviceStatsMonitor.on_train_batch_end/num_device_free"trainer/global_step2:JPO"@„e°b` =DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak"trainer/global_step2:JPmu<>m±bh EDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current"trainer/global_step2:JP&Ííf²ba >DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JPÔpÍZf³ba >DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JP7®†n´bi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JP'M8[rµbm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JP;‹úÆb¶b] :DeviceStatsMonitor.on_train_batch_end/active.all.allocated"trainer/global_step2:JPÿ Xf·ba >DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JP÷n´k¸bf CDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JPÿšÍIs¹bn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP¬Ž]pºbk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current"trainer/global_step2:JPˆÈ²nj»be BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated"trainer/global_step2:JP];i¼bd ADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current"trainer/global_step2:JPKY–Ðn½bi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JP·* i¾bd ADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed"trainer/global_step2:JPÙ”Œ`g¿bb ?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed"trainer/global_step2:JPö<œEjÀbe BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated"trainer/global_step2:JPÍÓºVÁbQ .DeviceStatsMonitor.on_train_batch_end/num_ooms"trainer/global_step2:JPÑ7”ˆjÂbe BDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current"trainer/global_step2:JPt …çcÃb^ ;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated"trainer/global_step2:JPbªPdÄb_ oÇbj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JP;¸?õ_ÈbZ 7DeviceStatsMonitor.on_train_batch_end/segment.all.freed"trainer/global_step2:JPT¥L~]ÉbX 5DeviceStatsMonitor.on_train_batch_end/active.all.peak"trainer/global_step2:JPV= iÊbd ADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated"trainer/global_step2:JPÙw••eËb` =DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed"trainer/global_step2:JPMÕ’ÅdÌb_ DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JPòúqyfòba >DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JPÀS1õqóbl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated"trainer/global_step2:JPpe>Õgôbb ?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current"trainer/global_step2:JPbÂwKkõbf CDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JP1½Ã eöb` =DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed"trainer/global_step2:JPºÙ„m÷bh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JPÙiaøb\ 9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak"trainer/global_step2:JPcàÎ_ùbZ 7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries"trainer/global_step2:JPŒ ßdgúbb ?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current"trainer/global_step2:JP,*ceûb` =DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak"trainer/global_step2:JPÂn•Çlübg DDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JP ƒ’ÿpýbk HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JPŠ”Š©\þbW 4DeviceStatsMonitor.on_train_batch_end/max_split_size"trainer/global_step2:JPåŒzÞgÿbb ?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak"trainer/global_step2:JP%:çr€bm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JP›h@Seb` =DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak"trainer/global_step2:JP•:ëyg‚bb ?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current"trainer/global_step2:JPI:¥Kdƒb_ DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JPcI=}^‰bY 6DeviceStatsMonitor.on_train_batch_end/segment.all.peak"trainer/global_step2:JPÛÎn jŠbe BDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JP™x©r‹bm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JP«õ%eŒb` =DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed"trainer/global_step2:JPïÆ¢¶pbk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JPX1lŽbg DDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated"trainer/global_step2:JPaÂ*~kbf CDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated"trainer/global_step2:JPÿäZfba >DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JPŸ4/âb‘b] :DeviceStatsMonitor.on_train_batch_end/allocation.all.freed"trainer/global_step2:JPÉø¸lk’bf CDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current"trainer/global_step2:JPaJWìj“be BDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JP xqDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 2021321692672 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1284 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 2244427589632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 2021339695616 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 99054920026 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚944514 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 2026709129850 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚49 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2297706 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 101421715968 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2297706 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚944514 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚357695 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 2244120665088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 1927654209824 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 2120429428736 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚559620 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚356998 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2322580 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 2021321692672 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 99340651008 % _timestamp‚1.7884640122074947e+09 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2322580 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚916618 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 325796352 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 101138731520 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 99107736064 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1353192 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 2026463116746 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582[At*•&¦M VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 1927654209824 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 2120429428736 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚559620 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚356998 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2322580 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 2021321692672 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 99340651008 % _timestamp‚1.7884640122074947e+09 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2322580 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚916618 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 325796352 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 101138731520 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 99107736064 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1353192 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 2026463116746 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 NDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 2021321692672 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1284 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 2244427589632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 2021339695616 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 99054920026 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚944514 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 2026709129850 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚49 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2297706 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 101421715968 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2297706 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚944514 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚357695 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 2244120665088y³-™A§b< loss/FBt4/assignment_loss"trainer/global_step2:JPõ¤A¨b< loss/FRt2/assignment_loss"trainer/global_step2:JP§¬ÅtC©b> loss/SRqqt1/assignment_loss"trainer/global_step2:JPä˜jû@ªb; loss/FRt4/detection_loss"trainer/global_step2:JPËÚ4°C«b> loss/SRqqt2/assignment_loss"trainer/global_step2:JP xï @¬b; loss/FBt1/detection_loss"trainer/global_step2:JP¢eÅ@­b; loss/FBt2/detection_loss"trainer/global_step2:JPÕõHm@®b; loss/FBt3/detection_loss"trainer/global_step2:JP¡¾ÄÂA¯b< loss/FBt1/assignment_loss"trainer/global_step2:JP©Óo²A°b< loss/FBt2/assignment_loss"trainer/global_step2:JP8\}6@±b; loss/FRt1/detection_loss"trainer/global_step2:JP(”1@²b; loss/FRt2/detection_loss"trainer/global_step2:JPäÚ¬bB³b= loss/SRqqt1/detection_loss"trainer/global_step2:JP²-5@´b; loss/FBt4/detection_loss"trainer/global_step2:JP!ÌCµb> loss/SRqqt4/assignment_loss"trainer/global_step2:JP¤+¢A¶b< loss/FBt3/assignment_loss"trainer/global_step2:JPÖ|fË@·b; loss/FRt3/detection_loss"trainer/global_step2:JPmðƒ«C¸b> loss/SRqqt3/assignment_loss"trainer/global_step2:JP«Œä.B¹b= loss/SRqqt2/detection_loss"trainer/global_step2:JPÓRóûBºb= loss/SRqqt3/detection_loss"trainer/global_step2:JPÚ1Oº-»b( epoch"trainer/global_step2:JP×åÿDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 4141696922112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 4384700722176 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 4141947826688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 285094912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 309035008 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 195592318464 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2ùï›'Ñ•O A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 4185496465920 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 199204256256 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 3961676194940 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚683 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚698986 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 3961430182044 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 4141696922112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 4384700722176 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 4141947826688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 285094912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 309035008 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 195592318464 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1839989 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 3946337505280 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2689298 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 4185520406016 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 trainer/global_step‚99 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 195592318464 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4529289 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 3766624684240 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2664452 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1088918 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1787904 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 _runtime‚ 123.355247244 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 3946337505280 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 3946355508224 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0-é5Ý0Ò+ _wandbruntime‚124  _runtime‚124P:¶±¡&Ó›M RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1099722 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1858264 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2716016 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 196992960534 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 197556357632 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 4227204376064 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 4183197902848 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1805341 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 3804172081024 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚705621 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 3985641545216 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 3985623542272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 4428376909824 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1099720 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚706309 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 _step‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 3985623542272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 309021696 trainer/global_step‚99 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 196763986790 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 4183197902848 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 3985641545216 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 4428685931520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 197323442688 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4549408 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 201196473856 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 197323442688 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 4182946984960 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4574282 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1858266 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 3804155041664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4574282 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚690 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 197556357632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2691144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 4227180435968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0ÄýôªÔŸM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 196992960534 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 197556357632 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 4227204376064 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 4183197902848 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1805341 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 3804172081024 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚705621 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 3985641545216 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 3985623542272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 4428376909824 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1099720 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚706309 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 _step‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 3985623542272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 309021696 trainer/global_step‚99 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 196763986790 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 4183197902848 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 3985641545216 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 4428685931520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 197323442688 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4549408 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 201196473856 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 197323442688 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 4182946984960 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4574282 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1858266 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 3804155041664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4574282 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚690 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 197556357632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2691144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 4227180435968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1099722 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1858264 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2716016 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496Wß+0Õ+ _wandbruntime‚124  _runtime‚124¹j"HÖ .loss/FRt4/detection_loss‚3.437301917403352 1loss/SRqqt3/detection_loss‚5.9462713118522394 _step‚7 0loss/SRqqt2/detection_loss‚4.985731347040697 /loss/FBt4/detection_loss‚10.962526086966196 /loss/FRt4/assignment_loss‚7.458775165034275 1loss/SRqqt1/assignment_loss‚5.311317614146641 $ _timestamp‚1.788464071214075e+09 0loss/FBt3/assignment_loss‚2.3406236993855445 /loss/FRt3/assignment_loss‚7.361598141988119 0loss/FBt2/assignment_loss‚2.5930780930952593 -loss/FRt3/detection_loss‚3.66730957892206 0loss/SRqqt1/detection_loss‚2.781729067143585 .loss/FBt1/detection_loss‚6.621794419288635 0loss/FBt1/assignment_loss‚2.6400370311737063 .loss/FRt1/detection_loss‚2.538440592159502 1loss/SRqqt4/detection_loss‚1.9465215721478064 /loss/FBt2/detection_loss‚3.4664639153916417 /loss/FRt1/assignment_loss‚7.193819491589656 .loss/FRt2/detection_loss‚5.366782959247991 .loss/FBt3/detection_loss‚5.674022532742599 trainer/global_step‚99 1loss/SRqqt3/assignment_loss‚5.486476621320171 /loss/FBt4/assignment_loss‚2.652847083409627 &loss/total_loss‚118.47141691294138 epoch‚0 _runtime‚ 124.398942389 /loss/FRt2/assignment_loss‚7.070183027358282 1loss/SRqqt2/assignment_loss‚5.422089302178585 1loss/SRqqt4/assignment_loss‚5.545676341955213òv‚L×Æ 0loss/SRqqt1/detection_loss‚2.781729067143585 .loss/FBt1/detection_loss‚6.621794419288635 0loss/FBt1/assignment_loss‚2.6400370311737063 .loss/FRt1/detection_loss‚2.538440592159502 1loss/SRqqt4/detection_loss‚1.9465215721478064 /loss/FBt2/detection_loss‚3.4664639153916417 /loss/FRt1/assignment_loss‚7.193819491589656 .loss/FRt2/detection_loss‚5.366782959247991 .loss/FBt3/detection_loss‚5.674022532742599 trainer/global_step‚99 1loss/SRqqt3/assignment_loss‚5.486476621320171 /loss/FBt4/assignment_loss‚2.652847083409627 &loss/total_loss‚118.47141691294138 epoch‚0 _runtime‚ 124.398942389 /loss/FRt2/assignment_loss‚7.070183027358282 1loss/SRqqt2/assignment_loss‚5.422089302178585 1loss/SRqqt4/assignment_loss‚5.545676341955213 .loss/FRt4/detection_loss‚3.437301917403352 1loss/SRqqt3/detection_loss‚5.9462713118522394 _step‚7 0loss/SRqqt2/detection_loss‚4.985731347040697 /loss/FBt4/detection_loss‚10.962526086966196 /loss/FRt4/assignment_loss‚7.458775165034275 1loss/SRqqt1/assignment_loss‚5.311317614146641 $ _timestamp‚1.788464071214075e+09 0loss/FBt3/assignment_loss‚2.3406236993855445 /loss/FRt3/assignment_loss‚7.361598141988119 0loss/FBt2/assignment_loss‚2.5930780930952593 -loss/FRt3/detection_loss‚3.66730957892206‹±’Ø‚(:‡ Ñ—çÔçè͇ gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚105.443' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚26.36075 gpu.0.smClock‚1200 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jþÞÜAÙ‚(:¶ Ñ—çÔùµê‰ disk.rbd2.in‚0! disk.nvme1n1.out‚ 1426.3515625 network.recv‚135275) disk./.usagePercent‚83.17280301062577 disk.md0.in‚ 0.42578125! disk.nvme0n1.out‚ 613.69921875 disk.sda1.in‚0 disk.nvme1n1.in‚ 38.234375 disk.sdb1.in‚0 disk.sda2.in‚ 0.2734375 disk.sdb2.in‚ 0.15234375 disk.md0.out‚14.15625 disk.dm-0.in‚ 208.00390625( proc.memory.percent‚17.8957462310791 disk.sdb2.out‚ 14.6435546875 disk.sda.in‚ 0.2734375 disk.dm-0.out‚ 2040.05078125 disk.md0p1.in‚ 0.42578125 network.sent‚178807 disk.sda.out‚ 14.6435546875 disk.sda1.out‚0 disk.sda2.out‚ 14.6435546875% memory_percent‚36.911797523498535* proc.memory.availableMB‚10336.37109375 disk.rbd2.out‚0% disk./.usageGB‚2974.6623039245605 disk.rbd0.out‚0 disk.sdb.in‚ 0.15234375 disk.sdb.out‚ 14.6435546875" proc.memory.rssMB‚ 2932.0390625 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.nvme0n1.in‚ 169.76953125 disk.md0p1.out‚14.15625 cpu‚23.57078330557652©ï¨Ä‘Ú‚(:† à—çÔÜÐ׆ gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚104.494' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚26.1235 gpu.0.smClock‚1215 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í©@\BÛ‚(:· à—çÔ³ýƒŠ disk.sdb2.in‚ 0.15234375 disk.md0.out‚ 16.2265625 disk.rbd0.out‚0 disk.sda2.in‚ 0.2734375) proc.memory.percent‚17.89577007293701 disk.sda1.in‚0 network.sent‚217339 disk.sdb2.out‚ 16.775390625 disk.sdb1.in‚0 disk.dm-0.out‚ 2229.2265625 disk.sdb.in‚ 0.15234375 disk.sda1.out‚0 disk.rbd2.out‚0 disk.sdb.out‚ 16.775390625 disk.nvme0n1.in‚ 169.76953125# proc.memory.rssMB‚ 2932.04296875) disk./.usagePercent‚83.15797687667926 disk.md0.in‚ 0.42578125 disk.sdb1.out‚0 proc.cpu.threads‚17 network.recv‚140519$ disk./.usageGB‚2974.132049560547 disk.md0p1.in‚ 0.42578125 disk.rbd0.in‚0 disk.sda2.out‚ 16.775390625* proc.memory.availableMB‚10336.03515625" disk.nvme1n1.out‚ 1559.45703125 disk.sda.in‚ 0.2734375 disk.sda.out‚ 16.775390625! disk.nvme0n1.out‚ 669.76953125 disk.dm-0.in‚ 208.00390625$ memory_percent‚36.91384792327881 cpu‚23.689003708757188 disk.md0p1.out‚ 16.2265625 disk.rbd2.in‚0 disk.nvme1n1.in‚ 38.234375¦qƥ܂(:„ ï—çԭφ gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚96.416' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚24.104 gpu.0.smClock‚1245 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñÍy—DÝ‚(:¹ ï—çÔ貈ˆ disk.md0p1.in‚ 0.42578125 disk.nvme0n1.out‚ 719.6015625 disk.dm-0.out‚2412.375 disk.sdb2.in‚ 0.15234375 disk.dm-0.in‚ 208.00390625 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda2.in‚ 0.2734375 disk.sdb2.out‚18.37255859375 disk.md0.in‚ 0.42578125 disk.rbd0.out‚0 disk.sdb.in‚ 0.15234375 disk.sdb1.out‚0 disk.nvme0n1.in‚ 169.76953125$ disk./.usageGB‚2974.131675720215 disk.rbd0.in‚0 disk.rbd2.in‚0! disk.nvme1n1.out‚ 1692.7734375$ memory_percent‚36.91418170928955* proc.memory.percent‚17.896080017089844 network.sent‚223243) disk./.usagePercent‚83.15796642394712* proc.memory.availableMB‚10335.98046875 proc.memory.rssMB‚ 2932.09375 network.recv‚144976 disk.sda1.in‚0 disk.sda2.out‚18.37255859375 disk.nvme1n1.in‚ 38.234375 disk.sdb.out‚18.37255859375 disk.sda.in‚ 0.2734375 disk.sda.out‚18.37255859375 cpu‚23.787363876413238 disk.md0.out‚ 17.76953125 disk.sdb1.in‚0 disk.md0p1.out‚ 17.76953125 disk.rbd2.out‚0á¬X‘Þ‚(:† þ—çÔíÇ͇ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚132.122' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.0305 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”CCÚDß‚(:¹ þ—çÔͦ„ˆ disk.md0.out‚ 19.109375% memory_percent‚36.916542053222656 disk.sdb.in‚ 0.15234375 disk.sdb1.in‚0 disk.sdb2.in‚ 0.15234375 disk.md0.in‚ 0.42578125 disk.rbd2.in‚0 disk.sda.out‚19.76318359375 disk.nvme0n1.in‚ 169.76953125' proc.memory.availableMB‚ 10335.59375 disk.md0p1.in‚ 0.42578125 disk.md0p1.out‚ 19.109375 network.sent‚230045) disk./.usagePercent‚83.15875400531631$ disk./.usageGB‚2974.159843444824 disk.sdb1.out‚0 disk.sda.in‚ 0.2734375 disk.rbd2.out‚0 network.recv‚149888" disk.nvme1n1.out‚ 1913.62109375 disk.sda1.in‚0* proc.memory.percent‚17.896127700805664 disk.nvme1n1.in‚ 38.234375 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0" proc.memory.rssMB‚ 2932.1015625 cpu‚23.874958301424563 proc.cpu.threads‚17 disk.dm-0.in‚ 208.00390625 disk.dm-0.out‚ 2633.84375 disk.sda2.in‚ 0.2734375 disk.sdb.out‚19.76318359375! disk.nvme0n1.out‚ 720.22265625 disk.sda2.out‚19.76318359375 disk.sdb2.out‚19.76318359375Jâìµ0à+ _wandbruntime‚182  _runtime‚182M’?Åá¿ ' lr-AdamW/pg2‚0.00010160748769183511 trainer/global_step‚149 % _timestamp‚1.7884641294980266e+09 _runtime‚ 182.682991665 _step‚8 ' lr-AdamW/pg1‚0.00010160748769183511Hcˆ¥Éâà ' lr-AdamW/pg1‚0.00010160748769183511 ' lr-AdamW/pg2‚0.00010160748769183511 trainer/global_step‚149 % _timestamp‚1.7884641294980266e+09 _runtime‚ 182.682991665 _step‚8åÿg0ã+ _wandbruntime‚182  _runtime‚182½¾Ãjš'ä”O K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2753739 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 293808025088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 299548839936 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 5910657357824 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 5910657357824 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1283 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 268317696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 5936132106648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4027250 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6269694968320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292257792 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2753741 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6756143 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 5910639354880 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2677085 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 5643142555440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚689 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1048067 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 6568975490560 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 6204465382912 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 6204214478336 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 6569267748352 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7884641294992223e+09 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2677774 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 292743538312 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 6204465382912 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1629018 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4027250 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1302 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6780991 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 293575123456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 5910639354880 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 6269718908416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 _runtime‚ 182.684518962 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 293575123456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚9 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2753739 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 299280522240 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 292972511848 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4002404 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1048754 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 293808025088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1629020 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2753741 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 6204214478336 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 5935886093752 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 trainer/global_step‚149 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4002404 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 293575123456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 5910639354880 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 6269718908416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 _runtime‚ 182.684518962 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 293575123456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚9 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2753739 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 299280522240 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 292972511848 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4002404 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1048754 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 293808025088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1629020 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2753741 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 6204214478336 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 5935886093752 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 trainer/global_step‚149 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4002404 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 6204214478336 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 6569267748352 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7884641294992223e+09 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2677774 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 292743538312 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 6204465382912 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1629018 ›on¹0æ+ _wandbruntime‚183  _runtime‚183}Á;ÂçžM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 5949925391872 M>DeviceStats3Fø“%Monitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 294702027298 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 6245464541184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2772014 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2772016 $ _timestamp‚1.788464130491265e+09 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 5975620953266 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2772014 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2695149 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306924544 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 294931001042 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6825984 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4029096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6801110 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _step‚10 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 6311402878464 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚149 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4029096 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6801110 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1055329 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1283 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6825984 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 6245715459072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚702 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2772016 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 295772064256 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 5949943394816 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 301557999616 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 NDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 5949943394816 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 301557999616 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 NDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 294702027298 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 6245464541184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2772014 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2772016 $ _timestamp‚1.788464130491265e+09 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 5975620953266 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2772014 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2695149 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306924544 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 294931001042 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6825984 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4029096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6801110 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _step‚10 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 6311402878464 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚149 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4029096 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6801110 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1055329 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1283 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6825984 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 6245715459072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚702 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 KZ0é+ _wandbruntime‚183  _runtime‚183KÜÌKêÅ /loss/FRt4/assignment_loss‚7.366585812917569 /loss/FRt3/detection_loss‚3.3274857216343587 -loss/FBt4/detection_loss‚12.2441932283915 .loss/FRt2/assignment_loss‚7.17348669634925 /loss/FBt2/assignment_loss‚0.967049852013588 &loss/total_loss‚105.23747872189709 _runtime‚ 183.678651623 0loss/FBt3/assignment_loss‚0.3838260586620594 0loss/SRqqt3/detection_loss‚4.258379313277035 /loss/FBt3/detection_loss‚3.1189155314041646 .loss/FRt1/assignment_loss‚7.13109389146169 /loss/FRt3/assignment_loss‚7.356911717628946 1loss/SRqqt1/detection_loss‚1.8897379648899941 trainer/global_step‚149 0loss/FBt4/assignment_loss‚0.9357609061094431 /loss/FBt2/detection_loss‚3.1143115310101868 _step‚11 /loss/FRt1/detection_loss‚2.5834395615383983 1loss/SRqqt2/assignment_loss‚5.042613477123027 0loss/FBt1/assignment_loss‚0.6107206407346224 .loss/FRt4/detection_loss‚4.311921710284745 1loss/SRqqt2/detection_loss‚3.2626696733795866 epoch‚0 1loss/SRqqt1/assignment_loss‚5.131266341064915 .loss/FBt1/detection_loss‚8.775445288733431 $ _timestamp‚1.788464130493935e+09 /loss/FRt2/detection_loss‚3.8847266361117363 1loss/SRqqt3/assignment_loss‚5.255233776278612 1loss/SRqqt4/assignment_loss‚5.416861281103017 0loss/SRqqt4/detection_loss‚1.694842109795213øÒª1OëÉ /loss/FRt1/detection_loss‚2.5834395615383983 1loss/SRqqt2/assignment_loss‚5.042613477123027 0loss/FBt1/assignment_loss‚0.6107206407346224 .loss/FRt4/detection_loss‚4.311921710284745 1loss/SRqqt2/detection_loss‚3.2626696733795866 epoch‚0 1loss/SRqqt1/assignment_loss‚5.131266341064915 .loss/FBt1/detection_loss‚8.775445288733431 $ _timestamp‚1.788464130493935e+09 /loss/FRt2/detection_loss‚3.8847266361117363 1loss/SRqqt3/assignment_loss‚5.255233776278612 1loss/SRqqt4/assignment_loss‚5.416861281103017 0loss/SRqqt4/detection_loss‚1.694842109795213 /loss/FRt4/assignment_loss‚7.366585812917569 /loss/FRt3/detection_loss‚3.3274857216343587 -loss/FBt4/detection_loss‚12.2441932283915 .loss/FRt2/assignment_loss‚7.17348669634925 /loss/FBt2/assignment_loss‚0.967049852013588 &loss/total_loss‚105.23747872189709 _runtime‚ 183.678651623 0loss/FBt3/assignment_loss‚0.3838260586620594 0loss/SRqqt3/detection_loss‚4.258379313277035 /loss/FBt3/detection_loss‚3.1189155314041646 .loss/FRt1/assignment_loss‚7.13109389146169 /loss/FRt3/assignment_loss‚7.356911717628946 1loss/SRqqt1/detection_loss‚1.8897379648899941 trainer/global_step‚149 0loss/FBt4/assignment_loss‚0.9357609061094431 /loss/FBt2/detection_loss‚3.1143115310101868 _step‚11 Ï&Ä‘ì‚(:† ˜çÔÒ‡‘ˆ gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚136.466' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.1165 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò—¸$í‚(:™ ˜çÔŠ´ðˆ disk.sdb2.in‚ 0.15234375 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sda.out‚21.625 disk.md0p1.in‚ 0.42578125 disk.md0p1.out‚ 20.9140625' proc.memory.availableMB‚ 10335.65625! proc.memory.rssMB‚ 2932.109375 network.sent‚247017 disk.md0.in‚ 0.42578125 disk.sda.in‚ 0.2734375 disk.dm-0.in‚ 208.00390625 disk.sda1.in‚0 disk.sda2.in‚ 0.2734375 disk.sda2.out‚21.625 disk.md0.out‚ 20.9140625 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.nvme1n1.out‚1975.625 disk.sdb1.in‚0) disk./.usagePercent‚83.14901387218347 disk.sdb2.out‚21.625 disk.rbd2.out‚0 disk.sdb.out‚21.625% disk./.usageGB‚2973.8114891052246 disk.nvme1n1.in‚ 38.234375 disk.sdb.in‚ 0.15234375 disk.dm-0.out‚ 2696.06640625 cpu‚23.948668943370325 disk.nvme0n1.in‚ 169.76953125! disk.nvme0n1.out‚ 720.44140625* proc.memory.percent‚17.896175384521484 proc.cpu.threads‚17 network.recv‚154740 disk.sda1.out‚0% memory_percent‚36.916160583496094=¤qöî‚(:… œ˜çÔñć‡ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚124.044' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.011 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e/ÃJï‚(:¿ œ˜çÔ»­¼‡ disk.sda.in‚ 0.2734375 disk.nvme0n1.in‚ 169.7734375 disk.nvme0n1.out‚ 720.4921875 disk.rbd0.out‚0) disk./.usagePercent‚83.15250999110536 disk.sda.out‚23.25244140625 disk.sda1.out‚0 disk.md0p1.in‚ 0.42578125 disk.sdb2.out‚23.25244140625 disk.md0.out‚ 22.4921875 disk.rbd0.in‚0 cpu‚24.015811546240293 disk.dm-0.in‚ 208.0078125 disk.md0p1.out‚ 22.4921875 disk.sdb2.in‚ 0.15234375 disk.md0.in‚ 0.42578125 disk.rbd2.in‚0 disk.nvme1n1.in‚ 38.234375* proc.memory.percent‚17.896199226379395 network.sent‚284978% disk./.usageGB‚2973.9365272521973 disk.sdb1.in‚0 disk.sda2.in‚ 0.2734375 network.recv‚159554 disk.sda2.out‚23.25244140625# proc.memory.rssMB‚ 2932.11328125 disk.sdb.out‚23.25244140625$ memory_percent‚36.91384792327881 disk.sdb.in‚ 0.15234375 disk.sdb1.out‚0 disk.dm-0.out‚ 2917.26953125 disk.sda1.in‚0 disk.rbd2.out‚0" disk.nvme1n1.out‚ 2196.77734375* proc.memory.availableMB‚10336.03515625 proc.cpu.threads‚17z³°j’ð‚(:‡ «˜çÔžœé‡ gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚138.919' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.72975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÒØIKEñ‚(:º «˜çÔ¿¬ˆ% disk./.usageGB‚2974.0615272521973 disk.sdb1.out‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda.out‚ 24.7822265625 disk.nvme0n1.in‚ 170.2890625 disk.sda1.in‚0 disk.sdb2.in‚ 0.15234375# proc.memory.rssMB‚ 2932.11328125* proc.memory.percent‚17.896199226379395 network.recv‚164033 disk.sda.in‚ 0.2734375 disk.sda1.out‚0 disk.md0.out‚ 23.97265625 disk.sdb.out‚ 24.7822265625$ memory_percent‚36.91558837890625$ proc.memory.availableMB‚10335.75 disk.sda2.in‚ 0.2734375 disk.nvme0n1.out‚ 720.7734375 disk.md0p1.out‚ 23.97265625" disk.nvme1n1.out‚ 2476.48828125 disk.dm-0.out‚ 3197.26171875 disk.md0p1.in‚ 0.42578125 disk.sdb2.out‚ 24.7822265625 disk.rbd0.out‚0 disk.sdb.in‚ 0.15234375 network.sent‚288703) disk./.usagePercent‚83.15600504342193 disk.dm-0.in‚ 212.56640625 disk.md0.in‚ 0.42578125 disk.rbd2.in‚0 disk.rbd2.out‚0 disk.nvme1n1.in‚ 42.27734375 cpu‚24.074714444319618 disk.sda2.out‚ 24.7822265625 disk.sdb1.in‚0q!ÿ’ò‚(:‡ º˜çÔٿˇ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚137.969' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.49225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðP‘Eó‚(:º º˜çÔË’ªˆ disk.sda2.in‚ 0.2734375 disk.rbd2.in‚0 disk.sda.out‚ 26.5810546875 disk.nvme0n1.in‚ 170.2890625! disk.nvme0n1.out‚ 720.82421875* proc.memory.percent‚17.896223068237305 cpu‚24.127316596487386 proc.cpu.threads‚17 network.sent‚297501% disk./.usageGB‚2974.3115272521973 disk.rbd0.out‚0 network.recv‚168956 disk.md0.in‚ 0.42578125 disk.md0.out‚25.71875 disk.rbd0.in‚0 disk.md0p1.out‚25.71875* proc.memory.availableMB‚10335.62109375 disk.sdb2.in‚ 0.15234375 disk.nvme1n1.in‚ 42.27734375 disk.sdb1.out‚0 disk.sda2.out‚ 26.5810546875 disk.sdb1.in‚0 disk.dm-0.in‚ 212.56640625 disk.dm-0.out‚ 3418.2421875" proc.memory.rssMB‚ 2932.1171875) disk./.usagePercent‚83.16299514805506 disk.rbd2.out‚0" disk.nvme1n1.out‚ 2697.41796875 disk.sdb.out‚ 26.5810546875 disk.sda1.out‚0% memory_percent‚36.916375160217285 disk.md0p1.in‚ 0.42578125 disk.sdb2.out‚ 26.5810546875 disk.sdb.in‚ 0.15234375 disk.sda.in‚ 0.2734375 disk.sda1.in‚0ÄùØÖ0ô+ _wandbruntime‚241  _runtime‚241–¡tbÆõÀ ' lr-AdamW/pg1‚0.00013570396007164554 ' lr-AdamW/pg2‚0.00013570396007164554 trainer/global_step‚199 % _timestamp‚1.7884641884236894e+09 _runtime‚ 241.608691899 _step‚12Ž’ÊöÄ _runtime‚ 241.608691899 _step‚12 ' lr-AdamW/pg1‚0.00013570396007164554 ' lr-AdamW/pg2‚0.00013570396007164554 trainer/global_step‚199 % _timestamp‚1.7884641884236894e+09 käSm0÷+ _wandbruntime‚241  _runtime‚241u[TPø•O L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _step‚13 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7910342005460 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3667489 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 8266982939136 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3569715 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304840704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 8753366946304 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 390910552356 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 % _timestamp‚1.7884641884244468e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 8266982939136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3667491 OFDeviceStatsMonitor.on_train_batch_start/inactive_sp‡ SmKlit.small_pool.peak‚1283 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3667489 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5340356 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 399754376192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280900608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2169120 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 8266732034560 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚669 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 392023731712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8353893470720 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚ 241.609541812 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 8353917410816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3667491 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 7874941204480 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 391790830080 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8266732034560 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 390681578820 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9032693 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5340356 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1399926 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7519660426640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104û,hŸ'ù™O SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2169118 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7910588018356 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚199 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 392023731712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3667491 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1283 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3667489 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5340356 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 399754376192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280900608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2169120 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 8266732034560 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚669 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 392023731712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8353893470720 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚ 241.609541812 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 8353917410816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3667491 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 7874941204480 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 391790830080 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8266732034560 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 390681578820 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9032693 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5340356 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1399926 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7519660426640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _step‚13 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7910342005460 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3667489 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 8266982939136 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3569715 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304840704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 8753366946304 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 390910552356 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 % _timestamp‚1.7884641884244468e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 8266982939136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5367048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9052812 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5391920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 401468861440 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 8308233015296 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 7914227241472 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 401753943040 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 392869041550 trainer/global_step‚199 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 393987770880 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 7914245244416 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1283 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 7557207823424 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 7914245244416 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 7914227241472 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1407730 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2179922 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 309021696 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3685766 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3685764 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1407009 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1302 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 7950076864974 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3586929 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 7557190784064 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 8308233015296 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2179920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 393754855936 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9077686 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5391920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 393754855936 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 8797046302208 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 % _timestamp‚1.7884641893941443e+09 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 393987770880 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3587652 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9077686 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚723 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚721 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285081600 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 NDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3685766 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 8797355323904 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 8307982097408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 392640067806 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 8308233015296 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 7914227241472 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 401753943040 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 392869041550 trainer/global_step‚199 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 393987770880 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 7914245244416 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1283 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 7557207823424 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 7914245244416 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 7914227241472 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1407730 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2179922 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 309021696 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3685766 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3685764 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1407009 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1302 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 7950076864974 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3586929 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 7557190784064 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 8308233015296 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2179920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 393754855936 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9077686 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5391920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 393754855936 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 8797046302208 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 % _timestamp‚1.7884641893941443e+09 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 393987770880 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3587652 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FxõƒG‰ :DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9077686 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚723 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚721 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285081600 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 NDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3685766 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 8797355323904 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 8307982097408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 392640067806 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816rÔx0ý+ _wandbruntime‚242  _runtime‚242Ç}‰³OþÉ .loss/FRt1/detection_loss‚2.189308063491531 /loss/FRt3/detection_loss‚3.8798253408500125 1loss/SRqqt1/detection_loss‚1.8265562188252806 0loss/SRqqt4/detection_loss‚3.167098601403011 _runtime‚ 242.581641158 /loss/FRt2/assignment_loss‚7.293090482552846 0loss/SRqqt3/detection_loss‚4.744346362778119 /loss/FBt3/detection_loss‚10.896454852819442 &loss/total_loss‚100.49583289493606 % _timestamp‚1.7884641893969154e+09 /loss/FRt3/assignment_loss‚7.378062588827951 0loss/FBt1/assignment_loss‚0.3918787689303587 0loss/SRqqt2/detection_loss‚2.695715899791634 /loss/FBt4/detection_loss‚10.415626915295919 2loss/SRqqt1/assignment_loss‚3.5637803808848063 1loss/SRqqt2/assignment_loss‚3.537588031668412 1loss/FBt3/assignment_loss‚0.15284962384030223 /loss/FBt2/detection_loss‚1.3357726559652463 /loss/FRt4/assignment_loss‚7.444344497862316 .loss/FRt4/detection_loss‚4.122150353022984 0loss/FBt4/assignment_loss‚0.8256629639305174 /loss/FBt1/detection_loss‚3.9131890929941284 _step‚15 trainer/global_step‚199 /loss/SRqqt3/assignment_loss‚4.2189720085689 .loss/FRt2/detection_loss‚4.483194291591647 epoch‚0 /loss/FRt1/assignment_loss‚7.210073574729588 1loss/SRqqt4/assignment_loss‚4.129486497710733 0loss/FBt2/assignment_loss‚0.6808048266003717D8Ü SÿÍ /loss/FBt3/detection_loss‚10.896454852819442 &loss/total_loss‚100.49583289493606 % _timestamp‚1.7884641893969154e+09 /loss/FRt3/assignment_loss‚7.378062588827951 0loss/FBt1/assignment_loss‚0.3918787689303587 0loss/SRqqt2/detection_loss‚2.695715899791634 /loss/FBt4/detection_loss‚10.415626915295919 2loss/SRqqt1/assignment_loss‚3.5637803808848063 1loss/SRqqt2/assignment_loss‚3.537588031668412 1loss/FBt3/assignment_loss‚0.15284962384030223 /loss/FBt2/detection_loss‚1.3357726559652463 /loss/FRt4/assignment_loss‚7.444344497862316 .loss/FRt4/detection_loss‚4.122150353022984 0loss/FBt4/assignment_loss‚0.8256629639305174 /loss/FBt1/detection_loss‚3.9131890929941284 _step‚15 trainer/global_step‚199 /loss/SRqqt3/assignment_loss‚4.2189720085689 .loss/FRt2/detection_loss‚4.483194291591647 epoch‚0 /loss/FRt1/assignment_loss‚7.210073574729588 1loss/SRqqt4/assignment_loss‚4.129486497710733 0loss/FBt2/assignment_loss‚0.6808048266003717 .loss/FRt1/detection_loss‚2.189308063491531 /loss/FRt3/detection_loss‚3.8798253408500125 1loss/SRqqt1/detection_loss‚1.8265562188252806 0loss/SRqqt4/detection_loss‚3.167098601403011 _runtime‚ 242.581641158 /loss/FRt2/assignment_loss‚7.293090482552846 0loss/SRqqt3/detection_loss‚4.744346362778119¾æ¬°’€‚(:‡ ɘçÔ‰±€ˆ gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚105.111' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚26.27775 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<|ÅJ‚(:¿ ɘçÔÌ€˜ˆ cpu‚24.171868072369012 disk.nvme1n1.in‚ 42.27734375 network.sent‚301720 disk.sda.in‚ 0.2734375 disk.rbd2.out‚0$ memory_percent‚36.92159652709961 network.recv‚174079$ disk./.usageGB‚2974.311496734619 disk.sda.out‚27.90185546875 disk.nvme0n1.in‚ 170.44921875 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sdb.out‚27.90185546875" proc.memory.rssMB‚ 2932.1171875 disk.dm-0.out‚ 3794.27734375 disk.md0p1.in‚ 0.42578125 disk.sdb2.out‚27.90185546875 disk.md0.out‚ 26.98828125 proc.cpu.threads‚17! disk.nvme0n1.out‚ 721.17578125 disk.md0p1.out‚ 26.98828125( proc.memory.availableMB‚ 10334.765625( disk./.usagePercent‚83.1629942947708 disk.sdb1.in‚0 disk.dm-0.in‚ 212.7265625 disk.md0.in‚ 0.42578125 disk.rbd2.in‚0* proc.memory.percent‚17.896223068237305 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sdb1.out‚0 disk.sda2.in‚ 0.2734375 disk.sda2.out‚27.90185546875 disk.sdb2.in‚ 0.15234375! disk.nvme1n1.out‚ 3073.1015625 disk.sdb.in‚ 0.15234375FÞ.(‚‚(:… ؘçÔ—–؇ gpu.0.gpu‚35.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚99.818' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚24.9545 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m/Cƒ‚(:¸ ؘçԽǾˆ' disk./.usagePercent‚83.160793461351 disk.nvme0n1.in‚ 170.44921875 network.sent‚353136 disk.sdb2.in‚ 0.15234375 disk.sdb2.out‚ 29.568359375% disk./.usageGB‚2974.2327842712402 disk.md0.out‚ 28.60546875 disk.sda.in‚ 0.2734375 disk.sda1.out‚0 disk.md0p1.in‚ 0.42578125 disk.rbd2.in‚0 disk.sdb.in‚ 0.15234375 disk.sdb1.in‚0 disk.sda.out‚ 29.568359375 disk.nvme0n1.out‚ 721.2265625* proc.memory.percent‚17.896223068237305 disk.md0p1.out‚ 28.60546875 disk.md0.in‚ 0.42578125 disk.rbd0.in‚0 disk.nvme1n1.in‚ 42.27734375" proc.memory.rssMB‚ 2932.1171875 disk.rbd2.out‚0" disk.nvme1n1.out‚ 3256.87890625 disk.dm-0.in‚ 212.7265625 cpu‚24.211939513218155 disk.sdb.out‚ 29.568359375 disk.sdb1.out‚0) proc.memory.availableMB‚ 10335.0078125 network.recv‚179202 disk.rbd0.out‚0 disk.dm-0.out‚ 3978.10546875 disk.sda1.in‚0 disk.sda2.in‚ 0.2734375 disk.sda2.out‚ 29.568359375$ memory_percent‚36.92011833190918 proc.cpu.threads‚17Xúü„‚(:… ç˜çÔ¡È̇ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚103.544' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.886 gpu.0.smClock‚1200 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ú¶€B…‚(:· ç˜çÔÞËŸˆ disk.sda1.out‚0 disk.md0p1.in‚ 0.42578125 disk.sdb2.out‚31.00439453125 disk.rbd2.out‚0 disk.sdb1.in‚0 disk.dm-0.in‚ 212.7265625 disk.sda2.out‚31.00439453125 proc.memory.rssMB‚2932.125 disk.sdb1.out‚0 disk.md0.in‚ 0.42578125 disk.sda2.in‚ 0.2734375 disk.md0p1.out‚ 29.9921875 disk.sdb2.in‚ 0.15234375 disk.md0.out‚ 29.9921875 network.recv‚183419 disk.sdb.out‚31.00439453125 disk.sda.out‚31.00439453125 disk.nvme0n1.out‚ 721.3984375 disk.dm-0.out‚ 4161.1015625 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚36.86072826385498 network.sent‚358716$ disk./.usageGB‚2973.781089782715 disk.rbd2.in‚0 cpu‚24.250678084972883 disk.nvme1n1.in‚ 42.27734375 disk.nvme1n1.out‚ 3439.703125 disk.sdb.in‚ 0.15234375 disk.sda.in‚ 0.2734375* proc.memory.availableMB‚10344.73828125 disk.sda1.in‚0* proc.memory.percent‚17.896270751953125 proc.cpu.threads‚17' disk./.usagePercent‚83.148163894403 disk.nvme0n1.in‚ 170.44921875ÆÏE’†‚(:‡ ö˜çÔö–†‡ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚103.267' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.81675 gpu.0.smClock‚1185 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03Ý ôB‡‚(:· ö˜çÔÑç—ˆ disk.sda1.in‚0 disk.sdb2.out‚32.75927734375 cpu‚24.285154120099275 proc.cpu.threads‚17 disk.sda.in‚ 0.2734375 disk.nvme0n1.in‚ 170.44921875$ proc.memory.availableMB‚10344.75 disk.md0.in‚ 0.42578125 disk.sdb.in‚ 0.15234375 disk.sda1.out‚0 disk.md0p1.in‚ 0.42578125 disk.nvme1n1.in‚ 42.27734375 disk.sdb1.in‚0 disk.sdb1.out‚0! disk.nvme0n1.out‚ 721.44921875$ memory_percent‚36.86065673828125* proc.memory.percent‚17.896270751953125 network.recv‚188345$ disk./.usageGB‚2973.730438232422 disk.rbd2.in‚0 disk.rbd2.out‚0 disk.dm-0.out‚ 4294.84765625 disk.sdb2.in‚ 0.15234375 proc.memory.rssMB‚2932.125( disk./.usagePercent‚83.1467476558575 disk.rbd0.in‚0 disk.md0p1.out‚ 31.69921875 disk.md0.out‚ 31.69921875 disk.rbd0.out‚0 disk.sdb.out‚32.75927734375 disk.sda2.in‚ 0.2734375 disk.sda2.out‚32.75927734375 network.sent‚365404! disk.nvme1n1.out‚ 3573.3984375 disk.sda.out‚32.75927734375 disk.dm-0.in‚ 212.7265625y6*Ù0ˆ+ _wandbruntime‚300  _runtime‚300gœ˜}‰¼ _step‚16 % lr-AdamW/pg1‚0.000169800432451456 % lr-AdamW/pg2‚0.000169800432451456 trainer/global_step‚249 % _timestamp‚1.7884642474300346e+09 _runtime‚ 300.615039386U"¯åÆŠÀ % lr-AdamW/pg1‚0.000169800432451456 % lr-AdamW/pg2‚0.000169800432451456 trainer/global_step‚249 % _timestamp‚1.7884642474300346e+09 _runtime‚ 300.615039386 _step‚16Ö+¡b0‹+ _wandbruntime‚300  _runtime‚300‡b6ƒ§'Œ¡O E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11259547 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1304 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚10937912062464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4581239 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 trainer/global_step‚249 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 9839261057024 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1285 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 9839261057024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 490006536704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6703154 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _step‚17 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚10329249590784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 287192064 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 % _timestamp‚1.7884642474313917e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2709218 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚659 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 499796149248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 9396178297840 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚10438115913216 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 499508957184 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11284395 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4458431 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 9396195337200 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 9839243054080 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚10937600930304 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚10329249590784 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 490239438336 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 488619619328 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 9885043930064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 300.616721823 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6678308 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1749872ÂH¸C&¥O XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 287192064 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 % _timestamp‚1.7884642474313917e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2709218 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚659 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 499796149248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 9396178297840 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚10438115913216 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 499508957184 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11284395 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4458431 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 9396195337200 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 9839243054080 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚10937600930304 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚10329249590784 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 490239438336 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 488619619328 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 9885043930064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 300.616721823 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6678308 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1749872 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11259547 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1304 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚10937912062464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4581239 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 trainer/global_step‚249 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 9839261057024 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1285 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 9839261057024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 490006536704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocÇqÓ hated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6703154 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _step‚17 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚10329249590784¨»¨Õ0Ž+ _wandbruntime‚301  _runtime‚301/™,¨°&ªM RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 492203477504 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚10479799883264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 490807082058 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚10370499653632 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11329388 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 491970562560 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4476804 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 9433725694624 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚10981622699520 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚249 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11329388 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4599516 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 501552414720 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 9433708655264 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 490578108314 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4599514 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6729872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1756146 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2720020 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2720022 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚10370750571520 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚10370750571520 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270401536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 9878529091072 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 491970562560 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4476166 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6705000 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚10981328357888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚638 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4599514 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4599516 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 9878547094016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1756782 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 9924532776682 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294341632 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚10479775943168 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 501822816256 % _timestamp‚1.7884642484740002e+09 _step‚18 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0ïí¸J´&®M GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚10370499653632 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11329388 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 491970562560 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4476804 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 9433725694624 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚10981622699520 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚249 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11329388 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4599516 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 501552414720 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 9433708655264 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 490578108314 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4599514 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6729872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1756146 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2720020 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2720022 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚10370750571520 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚10370750571520 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270401536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 9878529091072 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 491970562560 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4476166 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6705000 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚10981328357888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚638 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4599514 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4599516 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 9878547094016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1756782 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 9924532776682 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294341632 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚10479775943168 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 501822816256 % _timestamp‚1.7884642484740002e+09 _step‚18 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 492203477504 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚10479799883264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 490807082058ms¹Ì0‘+ _wandbruntime‚301  _runtime‚301¦‰K’Å /loss/FRt4/assignment_loss‚7.433467934528987 0loss/SRqqt3/detection_loss‚3.083975205725094 epoch‚0 trainer/global_step‚249 $ _timestamp‚1.788464248476581e+09 /loss/FRt2/assignment_loss‚7.217783553259713 /loss/SRqqt1/assignment_loss‚3.4608467279696 1loss/FBt3/assignment_loss‚0.23767262137912706 /loss/FRt3/assignment_loss‚7.360286746706281 0loss/FBt4/assignment_loss‚0.3387296505065428 .loss/FRt4/detection_loss‚3.411480992411574 1loss/SRqqt4/detection_loss‚1.9914928716330544 _runtime‚ 301.661372818 /loss/FRt1/assignment_loss‚7.139109802246094 .loss/FRt3/detection_loss‚3.641476371103809 .loss/FBt1/detection_loss‚3.086711487902159 1loss/SRqqt2/assignment_loss‚3.434981192482842 1loss/SRqqt3/assignment_loss‚3.777109056148889 2loss/SRqqt4/assignment_loss‚3.9329499618427173 0loss/FBt2/assignment_loss‚1.4122387766838074 .loss/FRt1/detection_loss‚2.308445216032366 .loss/FBt3/detection_loss‚4.408482278971111 0loss/SRqqt2/detection_loss‚3.536264987786611 /loss/FBt2/detection_loss‚13.567251278224427 &loss/total_loss‚103.08859405156102 _step‚19 1loss/FBt1/assignment_loss‚0.46627326131618413 .loss/FRt2/detection_loss‚6.946584850549698 0loss/SRqqt1/detection_loss‚2.393769245238646 .loss/FBt4/detection_loss‚8.501209980911678ºF®ÅO“É /loss/FRt2/assignment_loss‚7.217783553259713 /loss/SRqqt1/assignment_loss‚3.4608467279696 1loss/FBt3/assignment_loss‚0.23767262137912706 /loss/FRt3/assignment_loss‚7.360286746706281 0loss/FBt4/assignment_loss‚0.3387296505065428 .loss/FRt4/detection_loss‚3.411480992411574 1loss/SRqqt4/detection_loss‚1.9914928716330544 _runtime‚ 301.661372818 /loss/FRt1/assignment_loss‚7.139109802246094 .loss/FRt3/detection_loss‚3.641476371103809 .loss/FBt1/detection_loss‚3.086711487902159 1loss/SRqqt2/assignment_loss‚3.434981192482842 1loss/SRqqt3/assignment_loss‚3.777109056148889 2loss/SRqqt4/assignment_loss‚3.9329499618427173 0loss/FBt2/assignment_loss‚1.4122387766838074 .loss/FRt1/detection_loss‚2.308445216032366 .loss/FBt3/detection_loss‚4.408482278971111 0loss/SRqqt2/detection_loss‚3.536264987786611 /loss/FBt2/detection_loss‚13.567251278224427 &loss/total_loss‚103.08859405156102 _step‚19 1loss/FBt1/assignment_loss‚0.46627326131618413 .loss/FRt2/detection_loss‚6.946584850549698 0loss/SRqqt1/detection_loss‚2.393769245238646 .loss/FBt4/detection_loss‚8.501209980911678 /loss/FRt4/assignment_loss‚7.433467934528987 0loss/SRqqt3/detection_loss‚3.083975205725094 epoch‚0 trainer/global_step‚249 $ _timestamp‚1.788464248476581e+09ÚJ€”‚(:… …™çÔòÍ´† gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚102.658' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.6645 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²Rå’@•‚(:µ …™çÔЕ¦‰$ disk./.usageGB‚2973.809383392334 disk.sda.out‚34.17236328125 disk.md0p1.out‚ 33.05859375 disk.rbd2.in‚0 disk.rbd2.out‚0 disk.nvme1n1.out‚3706.5 disk.sda.in‚ 0.2734375 disk.nvme0n1.out‚ 802.3203125 disk.dm-0.in‚ 212.7265625 disk.md0.in‚ 0.42578125 network.sent‚369502 network.recv‚193209 disk.nvme0n1.in‚ 170.44921875 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sdb.in‚ 0.15234375 disk.sdb1.in‚0 proc.memory.rssMB‚2932.125 disk.sda1.out‚0 disk.sda2.out‚34.17236328125 disk.sdb2.in‚ 0.15234375* proc.memory.percent‚17.896270751953125) disk./.usagePercent‚83.14895499556975 disk.dm-0.out‚ 4508.8203125 disk.rbd0.in‚0 disk.sdb1.out‚0* proc.memory.availableMB‚10345.30078125 disk.sdb2.out‚34.17236328125 disk.nvme1n1.in‚ 42.27734375 cpu‚24.31648918783855 proc.cpu.threads‚17 disk.md0.out‚ 33.05859375 disk.sdb.out‚34.17236328125$ memory_percent‚36.85729503631592 disk.sda2.in‚ 0.2734375 disk.md0p1.in‚ 0.42578125H{lå’–‚(:‡ ”™çÔà³í† gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚138.919' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.72975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'NçnE—‚(:º ”™çÔì°÷‡ disk.dm-0.in‚ 212.7265625 disk.md0p1.in‚ 0.42578125 disk.sda.in‚ 0.2734375 disk.rbd2.in‚0 disk.dm-0.out‚ 4644.38671875$ memory_percent‚36.85436248779297 proc.cpu.threads‚17' proc.memory.availableMB‚ 10345.78125 disk.nvme0n1.in‚ 170.44921875 disk.nvme0n1.out‚ 804.84375 disk.rbd0.in‚0 disk.nvme1n1.in‚ 42.27734375 disk.sdb1.out‚0# proc.memory.rssMB‚ 2932.12890625* proc.memory.percent‚17.896294593811035 disk.sda.out‚ 35.7724609375 disk.sda1.out‚0 disk.sda2.in‚ 0.2734375 disk.md0p1.out‚ 34.61328125 disk.sdb.in‚ 0.15234375 disk.sdb1.in‚0 cpu‚24.34420790495988 network.recv‚198345 disk.sda2.out‚ 35.7724609375) disk./.usagePercent‚83.14989915460009% disk./.usageGB‚2973.8431510925293 disk.sda1.in‚0 disk.md0.in‚ 0.42578125 disk.rbd0.out‚0 disk.sdb.out‚ 35.7724609375 network.sent‚419454 disk.sdb2.in‚ 0.15234375 disk.sdb2.out‚ 35.7724609375 disk.md0.out‚ 34.61328125 disk.rbd2.out‚0" disk.nvme1n1.out‚ 3839.54296875î ¾ ’˜‚(:‡ £™çÔô¦‹‡ gpu.0.gpu‚42.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚143.263' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.81575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P>ÀL™‚(:Á £™çÔÜ disk.sda1.out‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2932.12890625 network.sent‚426627 disk.nvme0n1.in‚ 178.24609375 disk.rbd2.in‚0 disk.nvme1n1.in‚ 42.27734375 cpu‚24.370854884276845 disk.dm-0.out‚ 4865.890625 disk.rbd2.out‚0( disk./.usagePercent‚83.1525170307005 disk.md0p1.out‚ 36.60546875 disk.rbd0.out‚0 disk.sdb.in‚ 0.15234375 disk.sdb.out‚37.82666015625 proc.cpu.threads‚17 network.recv‚202539$ disk./.usageGB‚2973.936779022217 disk.sda.in‚ 0.2734375! disk.nvme0n1.out‚ 893.04296875 disk.md0p1.in‚ 0.42578125 disk.sdb2.out‚37.82666015625 disk.sda.out‚37.82666015625 disk.sda2.in‚ 0.2734375 disk.md0.out‚ 36.60546875" disk.nvme1n1.out‚ 3972.84765625$ memory_percent‚36.85281276702881* proc.memory.availableMB‚10346.03515625 disk.sda2.out‚37.82666015625 disk.md0.in‚ 0.42578125* proc.memory.percent‚17.896294593811035 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sdb2.in‚ 0.15234375 disk.sdb1.in‚0 disk.dm-0.in‚ 220.5234375K8z{0š+ _wandbruntime‚358  _runtime‚358?¨é@Æ›À ' lr-AdamW/pg1‚0.00020389690483126643 ' lr-AdamW/pg2‚0.00020389690483126643 trainer/global_step‚299 % _timestamp‚1.7884643056776347e+09 _runtime‚ 358.863484241 _step‚20œ“»/ÊœÄ ' lr-AdamW/pg1‚0.00020389690483126643 ' lr-AdamW/pg2‚0.00020389690483126643 trainer/global_step‚299 % _timestamp‚1.7884643056776347e+09 _runtime‚ 358.863484241 _step‚201d¾0+ _wandbruntime‚358  _runtime‚358"K)Gž©O LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3249318 _step‚21 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚11803562906624 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 588455144960 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 358.865662481 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 599740971520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13536097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚711 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 289289216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚12391767147008 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5347374 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3249320 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚11859499841772 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 % _timestamp‚1.7884643056799712e+09 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚12392018051584 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚11272713208400 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚12522314415616 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5494991 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 586786633372 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚11803562906624 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚12522290475520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5494991 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚11803544903680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 trainer/global_step‚299 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5494989 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚11803544903680 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚12391767147008 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 588222243328 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13511249 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚13122055387136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8041106 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚713 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 586557659836 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5346661 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8016260 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 313229312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚11859253828876 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚11272696169040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2098054 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 599451682304 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848EÑ$³'Ÿ­O G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13536097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚711 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 289289216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚12391767147008 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5347374 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3249320 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚11859499841772 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 % _timestamp‚1.7884643056799712e+09 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚12392018051584 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚11272713208400 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚12522314415616 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5494991 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 586786633372 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚11803562906624 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚12522290475520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5494991 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚11803544903680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 trainer/global_step‚299 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5494989 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚11803544903680 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚12391767147008 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 588222243328 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13511249 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚13122055387136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8041106 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚713 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 586557659836 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5346661 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8016260 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 313229312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚11859253828876 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚11272696169040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2098054 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 599451682304 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3249318 _step‚21 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚11803562906624 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 588455144960 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 358.865662481 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 599740971520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0¥H²;‘ ‚(:† ²™çÔÜ­Œˆ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.966' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.9915 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êŒ[æ<¡‚(:± ²™çÔ³ƈ* proc.memory.percent‚17.896294593811035 disk.md0.out‚ 37.921875 proc.cpu.threads‚17 network.recv‚207572$ disk./.usageGB‚2974.061721801758 disk.sdb2.out‚ 39.19140625 disk.rbd2.out‚0 disk.nvme1n1.in‚ 42.27734375 disk.sda.out‚ 39.19140625 disk.nvme0n1.in‚ 178.24609375 cpu‚24.396681767723898 network.sent‚444007 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚36.85731887817383 disk.rbd2.in‚0" disk.nvme1n1.out‚ 4105.89453125! disk.nvme0n1.out‚ 952.08203125 disk.dm-0.out‚ 5057.9765625 disk.sdb2.in‚ 0.15234375 disk.sdb.in‚ 0.15234375 disk.sdb1.out‚0 disk.dm-0.in‚ 220.5234375 disk.md0p1.in‚ 0.42578125 disk.md0p1.out‚ 37.921875 disk.sda2.out‚ 39.19140625 disk.rbd0.in‚0 disk.sdb.out‚ 39.19140625 disk.sda2.in‚ 0.2734375) disk./.usagePercent‚83.15601048310907 disk.md0.in‚ 0.42578125 disk.rbd0.out‚0 disk.sdb1.in‚0 disk.sda.in‚ 0.2734375( proc.memory.availableMB‚ 10345.296875# proc.memory.rssMB‚ 2932.12890625`Àf0¢+ _wandbruntime‚360  _runtime‚360ÕP#¹&£³M M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 360.022058812 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8042952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13581090 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚12433017209856 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 590419184128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5513266 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚13165425120768 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚299 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8067824 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚688 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚22 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚12563998385664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚686 % _timestamp‚1.7884643068363094e+09 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 590186269184 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚11310243565824 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 590186269184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚11842830940672 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚12433268127744 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚11842830940672 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13556216 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8067824 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 601733659648 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 588745122566 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5364092 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282984448 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚11310226526464 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5513266/ôP”þ¤·M ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚12563974445568 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 590419184128 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13581090 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚13165732045312 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚11898988688390 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚11842830940672 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚12433268127744 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚11842830940672 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13556216 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8067824 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 601733659648 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 588745122566 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5364092 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282984448 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚11310226526464 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5513266 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 360.022058812 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8042952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13581090 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚12433017209856 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 590419184128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5513266 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚13165425120768 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚299 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8067824 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚688 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚22 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚12563998385664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚686 % _timestamp‚1.7884643068363094e+09 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 590186269184 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚11310243565824 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 590186269184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ER§Ì .loss/FBt1/detection_loss‚4.276734554791212 epoch‚0 .loss/FRt2/assignment_loss‚7.33345420272262 /loss/FRt1/assignment_loss‚7.276491525038233 .loss/FRt3/detection_loss‚2.484512263127873 1loss/SRqqt4/detection_loss‚1.9935924485451266 _runtime‚ 360.024581963 _step‚23 /loss/FRt1/detection_loss‚3.0725137154870437 /loss/FRt4/assignment_loss‚7.357971858978272 1loss/SRqqt2/assignment_loss‚3.460646207133929 0loss/FBt2/assignment_loss‚0.7493074943444559 0loss/FBt3/assignment_loss‚0.1020568488365825 1loss/SRqqt2/detection_loss‚3.2420643977820873 /loss/FRt3/assignment_loss‚7.370100544344995 .loss/FRt2/detection_loss‚3.077092991934882 2loss/SRqqt1/assignment_loss‚3.3384750445683795 0loss/FBt1/assignment_loss‚0.4222047685933135 1loss/FBt4/assignment_loss‚0.14689488144611698 0loss/SRqqt3/detection_loss‚2.131643727313781 .loss/FBt2/detection_loss‚1.805791897695747 -loss/FBt3/detection_loss‚9.45012772269547 2loss/SRqqt3/assignment_loss‚3.4299857720084814 2loss/SRqqt4/assignment_loss‚3.8653703153133394 .loss/FBt4/detection_loss‚6.530952777551568 %loss/total_loss‚88.80374373240362 $ _timestamp‚1.788464306839737e+09 trainer/global_step‚299 /loss/FRt4/detection_loss‚4.1362830977886915 1loss/SRqqt1/detection_loss‚1.7494746743614087 oYLP¨‚(:Å Á™çÔ›¹Þˆ disk.sda1.in‚0 network.sent‚468250 disk.dm-0.in‚ 220.5234375 disk.sda2.in‚ 0.2734375 disk.sdb2.in‚ 0.15234375* proc.memory.percent‚17.896294593811035 cpu‚24.418220481537322) disk./.usagePercent‚83.16302906610427 disk.sda.out‚41.00927734375 disk.sdb1.in‚0 disk.nvme0n1.in‚ 178.24609375" disk.nvme0n1.out‚ 1011.23046875 disk.sda2.out‚41.00927734375 disk.md0p1.out‚ 39.67578125 disk.rbd0.out‚0 disk.nvme1n1.in‚ 42.27734375) proc.memory.availableMB‚ 10345.6328125% disk./.usageGB‚2974.3127403259277 disk.md0.out‚ 39.67578125 disk.rbd0.in‚0# proc.memory.rssMB‚ 2932.12890625 network.recv‚211970 disk.sdb2.out‚41.00927734375 disk.sdb.out‚41.00927734375 proc.cpu.threads‚17 disk.sda.in‚ 0.2734375 disk.sda1.out‚0 disk.md0p1.in‚ 0.42578125 disk.md0.in‚ 0.42578125 disk.rbd2.in‚0" disk.nvme1n1.out‚ 4239.41796875% memory_percent‚36.855268478393555 disk.sdb1.out‚0 disk.dm-0.out‚ 5250.6484375 disk.rbd2.out‚0 disk.sdb.in‚ 0.15234375Ž÷™’©‚(:‡ Á™çÔ³œ‰ gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚132.399' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.09975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0GïÑ™ª‚(:… ЙçÔóć gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚128.388' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.097 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õØ͘I«‚(:¾ ЙçÔ¥ôà‡$ disk./.usageGB‚2974.311721801758 disk.md0p1.in‚ 0.42578125 disk.sdb2.out‚ 42.4814453125 disk.rbd0.in‚0 disk.sdb.in‚ 0.15234375 disk.dm-0.in‚ 220.5234375 disk.dm-0.out‚ 5338.69921875# proc.memory.rssMB‚ 2932.12890625 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme1n1.out‚ 4239.515625 disk.sdb1.out‚0 disk.sda.out‚ 42.4814453125* proc.memory.percent‚17.896294593811035 cpu‚24.439527369896037 disk.md0.out‚ 41.08984375 network.sent‚509260 disk.md0p1.out‚ 41.08984375 disk.sdb2.in‚ 0.15234375 disk.nvme1n1.in‚ 42.27734375" disk.nvme0n1.out‚ 1099.18359375 disk.sda2.in‚ 0.2734375 disk.rbd0.out‚0 disk.sda.in‚ 0.2734375 disk.nvme0n1.in‚ 178.24609375$ memory_percent‚36.85257434844971* proc.memory.availableMB‚10346.07421875( disk./.usagePercent‚83.1630005877422 disk.rbd2.out‚0 disk.sdb.out‚ 42.4814453125 disk.sdb1.in‚0 disk.md0.in‚ 0.42578125 network.recv‚216719 disk.sda2.out‚ 42.4814453125 disk.rbd2.in‚0 proc.cpu.threads‚17½Kc8œ¬‚(:‘ ß™çÔ¼ë· gpu.0.gpu‚12.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚110.064' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚27.515999999999995 gpu.0.smClock‚1215 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OB­‚(:· ß™çÔ¯©Ðˆ cpu‚24.45974517323866 disk.sdb2.in‚ 0.15234375 disk.sdb1.out‚0 disk.nvme1n1.in‚ 42.27734375 disk.sdb.in‚ 0.15234375 disk.dm-0.in‚ 220.5234375 disk.sda1.in‚0 disk.sda2.out‚ 44.04296875 disk.rbd0.out‚0 disk.rbd2.out‚0 disk.sda2.in‚ 0.2734375 disk.sdb2.out‚ 44.04296875# proc.memory.rssMB‚ 2932.12890625) disk./.usagePercent‚83.16300282761337 disk.rbd2.in‚0* proc.memory.availableMB‚10346.06640625* proc.memory.percent‚17.896294593811035 network.sent‚514874% disk./.usageGB‚2974.3118019104004$ memory_percent‚36.85262203216553 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.nvme0n1.in‚ 178.24609375" disk.nvme0n1.out‚ 1187.44140625 disk.dm-0.out‚ 5560.39453125 disk.sda1.out‚0 disk.md0p1.in‚ 0.42578125 disk.sdb1.in‚0 network.recv‚220848 disk.nvme1n1.out‚ 4372.953125 disk.sda.out‚ 44.04296875 disk.md0.in‚ 0.42578125 disk.md0.out‚ 42.59765625 disk.sdb.out‚ 44.04296875 disk.sda.in‚ 0.2734375 disk.md0p1.out‚ 42.59765625~݈ó0®+ _wandbruntime‚417  _runtime‚417¤ž/ƯÀ _runtime‚ 417.326546916 _step‚24 ' lr-AdamW/pg1‚0.00023799337721107686 ' lr-AdamW/pg2‚0.00023799337721107686 trainer/global_step‚349 % _timestamp‚1.7884643641413743e+09¶í„üÊ°Ä ' lr-AdamW/pg1‚0.00023799337721107686 ' lr-AdamW/pg2‚0.00023799337721107686 trainer/global_step‚349 % _timestamp‚1.7884643641413743e+09 _runtime‚ 417.326546916 _step‚24»™ê0±+ _wandbruntime‚417  _runtime‚417kL¬[¯'²©O M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15787799 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6408741 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚13767864756224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 _runtime‚ 417.328416085 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14454284703232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚13767846753280 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9354212 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15762951 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9379058 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 684724673880 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884643641431093e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚14454284703232 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2441798 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚13833955753480 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 _step‚25 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6408739 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 686670851584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚705 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 686670851584 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6408741 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6230511 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 684495700344 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 699750796800 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 295580672 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 699455216128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚13149231079600 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3789418 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚349 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 686437949952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚13767864756224 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14454284703232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚13767846753280 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9354212 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15762951 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9379058 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 684724673880 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884643641431093e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚14454284703232 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2441798 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚13833955753480 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 _step‚25 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6408739 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 686670851584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚705 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 686670851584 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6408741 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6230511 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 684495700344 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 699750796800 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 295580672 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 699455216128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚13149231079600 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3789418 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚349 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 686437949952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0T¾U.0´+ _wandbruntime‚418  _runtime‚418#Æص³M S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚14495785683968 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚14495534766080 QEDeviceStatsMonitor.on_train_batch_endy¸#á%/allocation.large_pool.allocated‚6427016 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚13186761437024 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚15349935037440 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15807918 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9380904 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚14495785683968 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6427014 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚663 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 701738149376 trainer/global_step‚349 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15832792 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚ 418.523320251 _step‚26 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚665 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚13873198586994 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 688401975808 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚14648196888064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 686683163074 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15832792 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9405776 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6248121 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚15349630210048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 686454189330 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6427016 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6427014 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 688401975808 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3800222 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚13807132790272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 304827392 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9380904 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15807918 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 701457262080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚13807132790272 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 688634890752 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚14648172947968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚13873444600098 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚13807150793216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229Ùñúú½&¶·M M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚ 418.523320251 _step‚26 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚665 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚13873198586994 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 688401975808 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚14648196888064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 686683163074 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15832792 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9405776 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6248121 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚15349630210048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 686454189330 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6427016 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6427014 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 688401975808 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3800222 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚13807132790272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 304827392 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9380904 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15807918 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 701457262080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚13807132790272 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 688634890752 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚14648172947968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚13873444600098 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚13807150793216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚14495785683968 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚14495534766080 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6427016 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚13186761437024 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚15349935037440 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15807918 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9380904 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚14495785683968 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6427014 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚663 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 701738149376 trainer/global_step‚349 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15832792 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496û£Þ•0·+ _wandbruntime‚418  _runtime‚418ô(µñO¸É /loss/FRt4/assignment_loss‚7.340527795976208 1loss/FBt4/assignment_loss‚0.41613308858016834 1loss/SRqqt4/detection_loss‚2.4915775573643018 0loss/FBt1/assignment_loss‚0.2289692381678973 /loss/FRt3/detection_loss‚2.8750064226870355 0loss/SRqqt2/detection_loss‚3.028590287063636 /loss/FRt2/assignment_loss‚7.262460732460022 0loss/FBt2/assignment_loss‚0.8602081694101033 /loss/FRt1/detection_loss‚2.2407475148165035 1loss/SRqqt1/detection_loss‚1.4386568217797993 .loss/FBt4/detection_loss‚8.444468829561682 _step‚27 0loss/FBt3/assignment_loss‚0.1791903278297266 .loss/FBt1/detection_loss‚3.514312138661551 .loss/FBt2/detection_loss‚2.611650373143925 _runtime‚ 418.524605547 /loss/FRt3/assignment_loss‚7.319283522092379 2loss/SRqqt1/assignment_loss‚3.5124185158656194 2loss/SRqqt2/assignment_loss‚3.2926169002757355 .loss/FBt3/detection_loss‚7.809971346623367 $ _timestamp‚1.788464365339758e+09 /loss/FRt2/detection_loss‚3.7998741567134857 $loss/total_loss‚91.2666347604978 epoch‚0 trainer/global_step‚349 .loss/FRt1/assignment_loss‚7.23367650925167 2loss/SRqqt3/assignment_loss‚3.3377702832221985 2loss/SRqqt4/assignment_loss‚3.9107615030728855 -loss/FRt4/detection_loss‚5.05468162894249 1loss/SRqqt3/detection_loss‚3.0630810969353965YgFÚS¹Í /loss/FRt4/assignment_loss‚7.340527795976208 1loss/FBt4/assignment_loss‚0.41613308858016834 1loss/SRqqt4/detection_loss‚2.4915775573643018 0loss/FBt1/assignment_loss‚0.2289692381678973 /loss/FRt3/detection_loss‚2.8750064226870355 0loss/SRqqt2/detection_loss‚3.028590287063636 /loss/FRt2/assignment_loss‚7.262460732460022 0loss/FBt2/assignment_loss‚0.8602081694101033 /loss/FRt1/detection_loss‚2.2407475148165035 1loss/SRqqt1/detection_loss‚1.4386568217797993 .loss/FBt4/detection_loss‚8.444468829561682 _step‚27 0loss/FBt3/assignment_loss‚0.1791903278297266 .loss/FBt1/detection_loss‚3.514312138661551 .loss/FBt2/detection_loss‚2.611650373143925 _runtime‚ 418.524605547 /loss/FRt3/assignment_loss‚7.319283522092379 2loss/SRqqt1/assignment_loss‚3.5124185158656194 2loss/SRqqt2/assignment_loss‚3.2926169002757355 .loss/FBt3/detection_loss‚7.809971346623367 $ _timestamp‚1.788464365339758e+09 /loss/FRt2/detection_loss‚3.7998741567134857 $loss/total_loss‚91.2666347604978 epoch‚0 trainer/global_step‚349 .loss/FRt1/assignment_loss‚7.23367650925167 2loss/SRqqt3/assignment_loss‚3.3377702832221985 2loss/SRqqt4/assignment_loss‚3.9107615030728855 -loss/FRt4/detection_loss‚5.05468162894249 1loss/SRqqt3/detection_loss‚3.0630810969353965?ˆ‘º‚(:† î™çÔßïÓ‡ gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚101.986' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.4965 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ð*'¿I»‚(:¾ î™çÔÉáÛˆ network.sent‚519554 disk.dm-0.out‚ 5914.33203125 disk.md0p1.in‚ 0.42578125 disk.rbd0.out‚0 disk.nvme1n1.in‚ 42.27734375 disk.sda2.out‚45.84228515625 disk.md0.out‚ 44.36328125 network.recv‚225906 disk.sda1.in‚0 disk.md0p1.out‚ 44.36328125 disk.sdb2.out‚45.84228515625$ memory_percent‚36.85283660888672 cpu‚24.478163061476828$ disk./.usageGB‚2974.311725616455 disk.sdb2.in‚ 0.15234375 disk.sda1.out‚0 disk.sda2.in‚ 0.2734375 disk.rbd0.in‚0 disk.sdb.in‚ 0.15234375' proc.memory.availableMB‚ 10346.03125* proc.memory.percent‚17.896294593811035 proc.cpu.threads‚17 disk.dm-0.in‚ 220.5234375 disk.sdb.out‚45.84228515625 disk.sda.in‚ 0.2734375" disk.nvme0n1.out‚ 1275.39453125) disk./.usagePercent‚83.16300069440274 disk.md0.in‚ 0.42578125 disk.nvme1n1.out‚ 4638.9375 disk.sdb1.in‚0 disk.nvme0n1.in‚ 178.24609375# proc.memory.rssMB‚ 2932.12890625 disk.rbd2.in‚0 disk.rbd2.out‚0 disk.sdb1.out‚0 disk.sda.out‚45.84228515625ÚgÁ¼‚(:… ý™çÔÕúʇ gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚88.678' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚22.1695 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08•¾'½‚(:œ ý™çÔõŸ¬ˆ disk.sda1.out‚0% memory_percent‚36.853957176208496* proc.memory.percent‚17.896294593811035 disk.nvme1n1.in‚ 42.27734375 disk.sda.in‚ 0.2734375 disk.dm-0.out‚ 6002.59375 disk.sdb2.out‚47.25 cpu‚24.49598117315038 disk.md0.in‚ 0.42578125 disk.sdb1.in‚0! disk.nvme0n1.out‚ 1363.5078125 disk.dm-0.in‚ 220.5234375 disk.sda1.in‚0* proc.memory.availableMB‚10345.84765625 proc.cpu.threads‚17 disk.nvme0n1.in‚ 178.24609375 disk.sda2.in‚ 0.2734375# proc.memory.rssMB‚ 2932.12890625 network.sent‚570989 network.recv‚231119% disk./.usageGB‚2973.7815284729004 disk.sda2.out‚47.25 disk.sdb2.in‚ 0.15234375 disk.md0.out‚ 45.73046875 disk.rbd2.in‚0! disk.nvme1n1.out‚ 4639.0859375 disk.md0p1.in‚ 0.42578125 disk.md0p1.out‚ 45.73046875( disk./.usagePercent‚83.1481761603642 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sdb.in‚ 0.15234375 disk.sdb.out‚47.25 disk.rbd2.out‚0 disk.sdb1.out‚0 disk.sda.out‚47.25ßšn¾‚(:… ŒšçÔº»«‡ gpu.0.gpu‚39.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚106.052' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚26.513 gpu.0.smClock‚1215 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ·{K¿‚(:À Œšçԣ𪈠disk.sda1.out‚0% disk./.usageGB‚2973.7813148498535 disk.sda2.out‚48.76513671875 disk.sdb.in‚ 0.15234375 disk.sda.out‚48.76513671875 disk.dm-0.in‚ 220.5234375 disk.dm-0.out‚ 6185.0234375) proc.memory.availableMB‚ 10345.5703125 disk.sdb2.out‚48.76513671875 disk.sda1.in‚0$ memory_percent‚36.85564994812012 disk.md0.in‚ 0.42578125 disk.rbd2.out‚0 disk.sda.in‚ 0.2734375 network.recv‚235604 disk.md0p1.out‚ 47.19921875 disk.nvme1n1.in‚ 42.27734375 disk.sdb1.in‚0 disk.sda2.in‚ 0.2734375 disk.sdb2.in‚ 0.15234375 disk.md0.out‚ 47.19921875 disk.rbd0.in‚0! disk.nvme1n1.out‚ 4772.1328125 disk.nvme0n1.in‚ 178.24609375# proc.memory.rssMB‚ 2932.13671875* proc.memory.percent‚17.896342277526855 disk.rbd0.out‚0 disk.nvme0n1.out‚ 1412.890625 cpu‚24.51214096885087 disk.sdb.out‚48.76513671875 proc.cpu.threads‚17 disk.sdb1.out‚0 network.sent‚575118) disk./.usagePercent‚83.14817018737439 disk.md0p1.in‚ 0.42578125 disk.rbd2.in‚0T÷„HÀ‚(:½ ›šçÔÛ䔉 disk.sda.in‚ 0.2734375* proc.memory.percent‚17.896342277526855 disk.dm-0.out‚ 6318.9296875 disk.sda2.in‚ 0.2734375 network.sent‚582967$ disk./.usageGB‚2973.730739593506 disk.md0.in‚ 0.42578125 disk.nvme1n1.out‚ 4905.265625 disk.sda.out‚50.30322265625# proc.memory.rssMB‚ 2932.13671875 disk.sdb2.in‚ 0.15234375 disk.md0.out‚ 48.703125 disk.nvme0n1.in‚ 178.24609375! disk.nvme0n1.out‚ 1413.6640625 disk.dm-0.in‚ 220.5234375 disk.rbd2.out‚0* proc.memory.availableMB‚10345.78515625 cpu‚24.527741957576822) disk./.usagePercent‚83.14675608203953 disk.sdb1.out‚0 disk.sda1.out‚0 disk.sda2.out‚50.30322265625 disk.md0p1.in‚ 0.42578125 disk.sda1.in‚0 disk.sdb2.out‚50.30322265625 disk.rbd0.out‚0 disk.sdb.in‚ 0.15234375$ memory_percent‚36.85433864593506 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.nvme1n1.in‚ 42.27734375 disk.sdb.out‚50.30322265625 disk.md0p1.out‚ 48.703125 network.recv‚239697 disk.rbd0.in‚0 disk.sdb1.in‚0¹_2‘Á‚(:† ›šçÔÜ“¥‰ gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚136.466' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.1165 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âÝvÄ0Â+ _wandbruntime‚476  _runtime‚476«Ò‘Äþ & lr-AdamW/pg2‚0.0002720898495908873 trainer/global_step‚399 % _timestamp‚1.7884644229496005e+09 _runtime‚ 476.134576591 _step‚28 & lr-AdamW/pg1‚0.0002720898495908873o϶'ÈÄ & lr-AdamW/pg2‚0.0002720898495908873 trainer/global_step‚399 % _timestamp‚1.7884644229496005e+09 _runtime‚ 476.134576591 _step‚28 & lr-AdamW/pg1‚0.0002720898495908873˜ëh0Å+ _wandbruntime‚476  _runtime‚476ç8“ãáÆ­O L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7118390 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚16690687480320 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚15025748950800 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚17490571146752 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚685 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 782433740852 _runtime‚ 476.135805256 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚15732166605824 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚15732166605824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚15808165652292 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 784653656576 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10692164 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 784886558208 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2788185 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚‘oÒ0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18039501 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306937856 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚15025731911440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚16517053164032 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 784653656576 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282997760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚16516802259456 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _step‚29 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7322489 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 799576728576 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 trainer/global_step‚399 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 799859726336 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 782662714388 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚16516802259456 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚15732148602880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7322491 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920½ÉÜ—·'DZO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7118390 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚16690687480320 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚15025748950800 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚17490571146752 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚685 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 782433740852 _runtime‚ 476.135805256 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚15732166605824 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚15732166605824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚15808165652292 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 784653656576 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10692164 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 784886558208 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2788185 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18039501 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306937856 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚15025731911440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚16517053164032 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 784653656576 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282997760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚16516802259456 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _step‚29 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7322489 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 799576728576 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 trainer/global_step‚399 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 799859726336 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 782662714388 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚16516802259456 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚15732148602880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7322491 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909L,KK0È+ _wandbruntime‚477  _runtime‚477KØÀé¼&ɶM E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚15847654498702 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚17534250792448 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7340764 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18059620 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 $ _timestamp‚1.788464423893027e+09 _step‚30 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚15771434639872 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚15063262268864 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 784621203582 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚15771434639872 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7340764 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 784392229838 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚15771452642816 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚16558303240192 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 786617682432 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4340322 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10743728 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18084494 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7135303 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4340320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10718856 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 801855401984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 786850597376 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 trainer/global_step‚399 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7340766 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚689 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 278790144 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18059620 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2795670 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 302730240 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚17533948062208 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10718856 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚16732371450368 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚15063279308224 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104£ˆ‚chʺM K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18084494 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7135303 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4340320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10718856 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 801855401984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 786850597376 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 trainer/global_step‚399 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7340766 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚689 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 278790144 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18059620 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2795670 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 302730240 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚17533948062208 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10718856 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚16732371450368 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚15063279308224 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚15847654498702 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚17534250792448 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7340764 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18059620 J@DeviceStatsMonitor.on_tr7µ©Xain_batch_end/allocation.small_pool.peak‚30508 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 $ _timestamp‚1.788464423893027e+09 _step‚30 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚15771434639872 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚15063262268864 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 784621203582 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚15771434639872 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7340764 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 784392229838 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚15771452642816 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚16558303240192 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 786617682432 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4340322 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10743728 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ã1Àð0Ë+ _wandbruntime‚477  _runtime‚477ËõÿÒLÌÆ /loss/FBt4/assignment_loss‚0.835172303426169 .loss/FBt4/detection_loss‚5.234046035088026 %loss/total_loss‚98.05160319522285 _step‚31 .loss/FRt4/detection_loss‚4.171381978525056 1loss/SRqqt3/detection_loss‚2.5551733083108017 .loss/FBt1/detection_loss‚0.881926369845818 /loss/FBt3/detection_loss‚12.241811123761265 $ _timestamp‚1.788464423895453e+09 /loss/FRt3/assignment_loss‚7.388616311348091 /loss/FRt4/assignment_loss‚7.430480413966709 0loss/FBt1/assignment_loss‚0.6217440249650211 1loss/FBt3/assignment_loss‚0.41994790840677026 2loss/SRqqt3/assignment_loss‚3.2122745472088194 -loss/FRt2/detection_loss‚6.70551763687815 _runtime‚ 477.080180361 .loss/FRt1/detection_loss‚2.164085912608331 0loss/SRqqt4/detection_loss‚3.713117391870516 1loss/SRqqt2/assignment_loss‚3.390534356788353 2loss/SRqqt4/assignment_loss‚3.0983131280759486 trainer/global_step‚399 /loss/FRt1/assignment_loss‚7.152319931214856 /loss/FRt2/assignment_loss‚7.162089007241385 2loss/SRqqt1/assignment_loss‚3.3467208520690006 .loss/FRt3/detection_loss‚2.541917836262008 1loss/FBt2/assignment_loss‚0.40431154872640035 1loss/SRqqt1/detection_loss‚1.3671833450943771 /loss/SRqqt2/detection_loss‚5.35662904492131 .loss/FBt2/detection_loss‚6.656288878619671 epoch‚0Ó:PÍÊ 1loss/SRqqt1/detection_loss‚1.3671833450943771 /loss/SRqqt2/detection_loss‚5.35662904492131 .loss/FBt2/detection_loss‚6.656288878619671 epoch‚0 /loss/FBt4/assignment_loss‚0.835172303426169 .loss/FBt4/detection_loss‚5.234046035088026 %loss/total_loss‚98.05160319522285 _step‚31 .loss/FRt4/detection_loss‚4.171381978525056 1loss/SRqqt3/detection_loss‚2.5551733083108017 .loss/FBt1/detection_loss‚0.881926369845818 /loss/FBt3/detection_loss‚12.241811123761265 $ _timestamp‚1.788464423895453e+09 /loss/FRt3/assignment_loss‚7.388616311348091 /loss/FRt4/assignment_loss‚7.430480413966709 0loss/FBt1/assignment_loss‚0.6217440249650211 1loss/FBt3/assignment_loss‚0.41994790840677026 2loss/SRqqt3/assignment_loss‚3.2122745472088194 -loss/FRt2/detection_loss‚6.70551763687815 _runtime‚ 477.080180361 .loss/FRt1/detection_loss‚2.164085912608331 0loss/SRqqt4/detection_loss‚3.713117391870516 1loss/SRqqt2/assignment_loss‚3.390534356788353 2loss/SRqqt4/assignment_loss‚3.0983131280759486 trainer/global_step‚399 /loss/FRt1/assignment_loss‚7.152319931214856 /loss/FRt2/assignment_loss‚7.162089007241385 2loss/SRqqt1/assignment_loss‚3.3467208520690006 .loss/FRt3/detection_loss‚2.541917836262008 1loss/FBt2/assignment_loss‚0.40431154872640035šA‚’΂(:‡ ªšçÔ„¶À‡ gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚138.919' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.72975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bŽËjBÏ‚(:· ªšçÔÙꙈ# proc.memory.rssMB‚ 2932.13671875 disk.sda1.out‚0 disk.sdb.out‚52.12060546875 disk.sda.out‚52.12060546875% disk./.usageGB‚2973.8094749450684 disk.dm-0.out‚ 6599.15625 disk.sda2.in‚ 0.2734375 disk.rbd2.in‚0 disk.nvme1n1.out‚ 5104.78125 disk.dm-0.in‚ 220.5234375 disk.md0.out‚ 50.4609375 proc.cpu.threads‚17 network.recv‚244580 disk.sda1.in‚0 disk.nvme1n1.in‚ 42.27734375 disk.sda.in‚ 0.2734375 disk.nvme0n1.in‚ 178.24609375 disk.nvme0n1.out‚1494.375 cpu‚24.54212380506281) disk./.usagePercent‚83.14895755542253 disk.md0p1.in‚ 0.42578125 disk.sdb2.out‚52.12060546875 disk.rbd0.in‚0 disk.sdb.in‚ 0.15234375$ memory_percent‚36.85612678527832* proc.memory.percent‚17.896342277526855 disk.sdb2.in‚ 0.15234375 disk.sdb1.in‚0 disk.sdb1.out‚0 network.sent‚600281 disk.sda2.out‚52.12060546875 disk.md0p1.out‚ 50.4609375 disk.md0.in‚ 0.42578125 disk.rbd0.out‚0 disk.rbd2.out‚0) proc.memory.availableMB‚ 10345.4921875Q¦dy’Ђ(:‡ ¹šçÔ⇇ gpu.0.gpu‚27.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.787' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚27.44675 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BsÔHÑ‚(:½ ¹šçÔ皈 disk.md0.in‚ 0.42578125 proc.cpu.threads‚17" disk.nvme1n1.out‚ 5238.15234375) disk./.usagePercent‚83.14989990122382 disk.md0.out‚ 51.89453125 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sdb1.in‚0 disk.md0p1.out‚ 51.89453125$ memory_percent‚36.85424327850342 network.sent‚638803 disk.sdb.in‚ 0.15234375 disk.nvme0n1.in‚ 178.24609375 disk.sda2.out‚ 53.6025390625 disk.md0p1.in‚ 0.42578125* proc.memory.availableMB‚10345.80078125 disk.sdb2.out‚ 53.6025390625 disk.nvme1n1.in‚ 42.27734375 disk.sda1.in‚0* proc.memory.percent‚17.896342277526855 disk.sdb2.in‚ 0.15234375 disk.rbd2.out‚0 disk.sdb.out‚ 53.6025390625 disk.sda.in‚ 0.2734375" disk.nvme0n1.out‚ 1497.25390625 disk.dm-0.out‚ 6735.40625 disk.sda1.out‚0 cpu‚24.555614008185756# disk./.usageGB‚2973.84317779541 disk.sda.out‚ 53.6025390625 disk.sda2.in‚ 0.2734375# proc.memory.rssMB‚ 2932.13671875 network.recv‚249634 disk.dm-0.in‚ 220.5234375 disk.rbd0.in‚0 disk.sdb1.out‚0ÚÚcÂ’Ò‚(:‡ ÈšçÔ¦¥è‡ gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚139.251' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.81275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«Á¢ÆBÓ‚(:· ÈšçÔè—ƒˆ network.sent‚643176 network.recv‚254551$ disk./.usageGB‚2973.936794281006 disk.sdb1.out‚0 disk.sda.in‚ 0.2734375 disk.dm-0.out‚6956.5) disk./.usagePercent‚83.15251745734261 disk.sdb1.in‚0 disk.sda.out‚54.97314453125 disk.sdb2.out‚54.97314453125 disk.rbd2.in‚0 disk.sdb.in‚ 0.15234375 disk.nvme0n1.in‚ 274.3828125 disk.dm-0.in‚ 352.0078125 disk.sda2.out‚54.97314453125 disk.sdb2.in‚ 0.15234375 disk.md0.out‚ 53.22265625 disk.rbd0.out‚0* proc.memory.percent‚17.896342277526855" disk.nvme0n1.out‚ 1585.26953125 disk.sda1.out‚0 disk.rbd0.in‚0$ memory_percent‚36.85576915740967* proc.memory.availableMB‚10345.55078125 cpu‚24.56834042422701" disk.nvme1n1.out‚ 5371.23046875 disk.md0.in‚ 0.42578125 disk.rbd2.out‚0 disk.nvme1n1.in‚77.625 disk.sda1.in‚0 disk.md0p1.out‚ 53.22265625 disk.sdb.out‚54.97314453125 disk.sda2.in‚ 0.2734375 disk.md0p1.in‚ 0.42578125# proc.memory.rssMB‚ 2932.13671875 proc.cpu.threads‚17ó‹¶ä‘Ô‚(:† ךçÔÿ€Â‡ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚138.302' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.5755 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mI´BÕ‚(:· ךçÔ÷®õ‡ disk.sdb1.out‚0 disk.sda1.in‚0! disk.nvme1n1.out‚ 5504.3984375 disk.dm-0.out‚ 7177.8125 disk.md0.in‚ 0.42578125$ memory_percent‚36.85588836669922 proc.cpu.threads‚17 disk.md0.out‚ 55.06640625 disk.dm-0.in‚ 361.48046875 disk.md0p1.in‚ 0.42578125 disk.md0p1.out‚ 55.06640625 disk.rbd0.in‚0% disk./.usageGB‚2974.0617637634277 disk.sdb1.in‚0 disk.nvme0n1.in‚ 283.85546875 network.recv‚258527 disk.sda2.out‚56.86474609375 disk.sdb2.out‚56.86474609375! proc.memory.rssMB‚ 2932.140625 disk.sda.out‚56.86474609375* proc.memory.percent‚17.896366119384766 cpu‚24.580841371255243 network.sent‚651006) disk./.usagePercent‚83.15601165637491 disk.nvme1n1.in‚77.625 disk.sdb.in‚ 0.15234375 disk.sdb.out‚56.86474609375 disk.sda.in‚ 0.2734375! disk.nvme0n1.out‚ 1673.4140625 disk.sda2.in‚ 0.2734375 disk.sdb2.in‚ 0.15234375 disk.rbd2.out‚0 disk.sda1.out‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 10345.53125 disk.rbd2.in‚0ÅO0Ö+ _wandbruntime‚535  _runtime‚535è9:Ä×¾ trainer/global_step‚449 % _timestamp‚1.7884644818472657e+09 _runtime‚ 535.032257458 _step‚32 & lr-AdamW/pg1‚0.0003061863219706978 & lr-AdamW/pg2‚0.0003061863219706978ÄÀVèÈØ _step‚32 & lr-AdamW/pg1‚0.0003061863219706978 & lr-AdamW/pg2‚0.0003061863219706978 trainer/global_step‚449 % _timestamp‚1.7884644818472657e+09 _runtime‚ 535.032257458 .4-0Ù+ _wandbruntime‚535  _runtime‚535îÐÁ‘²'Ú¬O PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12030116 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 882869363200 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚17696450452480 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚19674483127808 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 899597145088 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚33 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 535.033552113 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19674779579904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚16902249782640 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12054962 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚18579570720256 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8005974 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 882869363200 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚17696450452480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3137060 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8236239 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3136356 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18774885982720 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚18579319815680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296452096 trainer/global_step‚449 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 880600754896 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8236241 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12030116 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20291203 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8236241 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8236239 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 883102264832 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4869620 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 272512000 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚18579319815680 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 899869657088 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚17696468455424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 $ _timestamp‚1.788464481848389e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚17696468455424 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12054962 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚18579570720256 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 883102264832 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8006680 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 880371781360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚17782867576896 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚16902266822000”)úëk%Û°O H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19674779579904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚16902249782640 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12054962 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚18579570720256 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8005974 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 882869363200 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚17696450452480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3137060 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8236239 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3136356 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18774885982720 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚18579319815680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296452096 trainer/global_step‚449 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 880600754896 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8236241 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12030116 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20291203 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8236241 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8236239 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 883102264832 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4869620 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 272512000 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚18579319815680 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 899869657088 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚17696468455424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 $ _timestamp‚1.788464481848389e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚17696468455424 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12054962 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚18579570720256 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 883102264832 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8006680 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 880371781360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚17782867576896 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚16902266822000 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12030116 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 882869363200 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚17696450452480 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚19674483127808 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 899597145088 O@DeviceStatsMonitor.onÜc}K_train_batch_start/requested_bytes.all.peak‚ 6741051909 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚33 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 535.033552113!¶Ù‡0Ü+ _wandbruntime‚536  _runtime‚536w˜½&Ý·M NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12081680 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20336196 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4880422 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚16939797179424 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 901874500096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚17822110410410 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8254516 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚18620820796416 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285081600 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20311322 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12056808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 trainer/global_step‚449 _step‚34 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚17735754492416 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8254516 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚17735736489472 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚20311322 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884644829441013e+09 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8023995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 885066304000 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12056808 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 901589418496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚19718159371264 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚16939780140064 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 _runtime‚ 536.130862791 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12081680 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3143575 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8254514 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 884833389056 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19718468392960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚18620569878528 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 884833389056 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608ÄŠ UÁ&Þ»M ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12081680 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3143575 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8254514 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 884833389056 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19718468392960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚18620569878528 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 884833389056 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12081680 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20336196 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4880422 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚16939797179424 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 901874500096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚17822110410410 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8254516 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚18620820796416 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285081600 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20311322 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12056808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 trainer/global_step‚449 _step‚34 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚17735754492416 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8254516 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚17735736489472 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚20311322 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884644829441013e+09 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8023995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 885066304000 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12056808 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 901589418496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚19718159371264 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚16939780140064 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 _runtime‚ 536.130862791 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582"Ä »0ß+ _wandbruntime‚536  _runtime‚536ƒTŸºQàË trainer/global_step‚449 % _timestamp‚1.7884644829475155e+09 1loss/FBt2/assignment_loss‚0.43155260337807705 .loss/FRt2/detection_loss‚3.782985760105981 1loss/SRqqt2/detection_loss‚3.4526293754577635 1loss/SRqqt1/assignment_loss‚2.990503748257955 2loss/SRqqt4/assignment_loss‚3.3494128474482783 /loss/FRt1/detection_loss‚2.7533717020242303 /loss/FRt4/detection_loss‚3.4341071120330264 .loss/FBt3/detection_loss‚4.076847490668297 epoch‚0 /loss/FRt3/assignment_loss‚5.705223594393049 0loss/FBt1/assignment_loss‚0.7228937001457192 .loss/FBt2/detection_loss‚5.942299148440361 _runtime‚ 536.132749589 /loss/FRt2/assignment_loss‚7.376617617077297 0loss/FRt4/assignment_loss‚7.3167232899438766 2loss/SRqqt2/assignment_loss‚3.1940093278884887 1loss/FBt3/assignment_loss‚0.08701000785590622 /loss/FRt3/detection_loss‚2.4890127503682695 1loss/SRqqt1/detection_loss‚1.7077915879094125 $loss/total_loss‚84.4128724734257 /loss/FRt1/assignment_loss‚7.216387183577926 1loss/SRqqt3/assignment_loss‚3.377415866851807 0loss/FBt4/assignment_loss‚0.2933610254138544 -loss/FBt4/detection_loss‚7.61571244281881 _step‚35 1loss/SRqqt3/detection_loss‚1.8931213491161665 0loss/SRqqt4/detection_loss‚2.535247285056997 .loss/FBt1/detection_loss‚2.668635657194135­vÎòUáÏ /loss/FRt4/detection_loss‚3.4341071120330264 .loss/FBt3/detection_loss‚4.076847490668297 epoch‚0 /loss/FRt3/assignment_loss‚5.705223594393049 0loss/FBt1/assignment_loss‚0.7228937001457192 .loss/FBt2/detection_loss‚5.942299148440361 _runtime‚ 536.132749589 /loss/FRt2/assignment_loss‚7.376617617077297 0loss/FRt4/assignment_loss‚7.3167232899438766 2loss/SRqqt2/assignment_loss‚3.1940093278884887 1loss/FBt3/assignment_loss‚0.08701000785590622 /loss/FRt3/detection_loss‚2.4890127503682695 1loss/SRqqt1/detection_loss‚1.7077915879094125 $loss/total_loss‚84.4128724734257 /loss/FRt1/assignment_loss‚7.216387183577926 1loss/SRqqt3/assignment_loss‚3.377415866851807 0loss/FBt4/assignment_loss‚0.2933610254138544 -loss/FBt4/detection_loss‚7.61571244281881 _step‚35 1loss/SRqqt3/detection_loss‚1.8931213491161665 0loss/SRqqt4/detection_loss‚2.535247285056997 .loss/FBt1/detection_loss‚2.668635657194135 trainer/global_step‚449 % _timestamp‚1.7884644829475155e+09 1loss/FBt2/assignment_loss‚0.43155260337807705 .loss/FRt2/detection_loss‚3.782985760105981 1loss/SRqqt2/detection_loss‚3.4526293754577635 1loss/SRqqt1/assignment_loss‚2.990503748257955 2loss/SRqqt4/assignment_loss‚3.3494128474482783 /loss/FRt1/detection_loss‚2.7533717020242303#8ÆaÞ’â‚(:‡ æšçÔÍæû‡ gpu.0.gpu‚24.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚103.267' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.81675 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a§ôAã‚(:¶ æšçÔ‹Ë»ˆ disk.sda1.out‚0 network.sent‚668320 network.recv‚263284) disk./.usagePercent‚83.16304357193664 disk.nvme1n1.in‚77.625 disk.sda.in‚ 0.27734375 disk.dm-0.out‚ 7310.8984375* proc.memory.percent‚17.896366119384766 disk.sdb1.in‚0 disk.sdb2.in‚ 0.15234375 disk.sdb2.out‚58.36865234375 disk.md0.in‚ 0.4296875 disk.rbd0.out‚0' proc.memory.availableMB‚ 10345.71875 disk.sda2.in‚ 0.27734375 cpu‚24.59219466055989 proc.cpu.threads‚17 disk.sdb.out‚58.36865234375 disk.md0p1.in‚ 0.4296875 disk.md0p1.out‚ 56.515625 disk.rbd0.in‚0$ memory_percent‚36.85474395751953 disk.sda1.in‚0 disk.sda2.out‚58.36865234375$ disk./.usageGB‚2974.313259124756" disk.nvme0n1.out‚ 1673.46484375" disk.nvme1n1.out‚ 5637.43359375 disk.sdb.in‚ 0.15234375 disk.sdb1.out‚0! proc.memory.rssMB‚ 2932.140625 disk.sda.out‚58.36865234375 disk.md0.out‚ 56.515625 disk.rbd2.in‚0 disk.rbd2.out‚0 disk.nvme0n1.in‚ 283.85546875 disk.dm-0.in‚ 361.48046875xFŸ’ä‚(:‡ õšçÔ×—Þ‡ gpu.0.gpu‚37.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚105.111' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚26.27775 gpu.0.smClock‚1230 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ѹIFå‚(:» õšçÔ­¿Þˆ network.sent‚706888 disk.md0.out‚ 58.03515625 disk.rbd0.out‚0 disk.nvme1n1.in‚77.625 disk.dm-0.out‚ 7589.1640625 disk.md0p1.out‚ 58.03515625$ memory_percent‚36.83934211730957) disk./.usagePercent‚83.16305914437433 disk.sdb2.in‚ 0.15234375 disk.sdb.in‚ 0.15234375 disk.sdb1.in‚0 disk.nvme0n1.in‚ 283.85546875 disk.dm-0.in‚ 361.48046875 disk.sda1.out‚0 disk.md0p1.in‚ 0.4296875 network.recv‚268610* proc.memory.percent‚17.896366119384766% disk./.usageGB‚2974.3138160705566 disk.nvme1n1.out‚ 5771.140625 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sda2.out‚59.94970703125 proc.cpu.threads‚17 disk.sdb2.out‚59.94970703125 disk.sda.in‚ 0.27734375 disk.sda.out‚59.94970703125) proc.memory.availableMB‚ 10348.2421875 disk.md0.in‚ 0.4296875 disk.rbd2.in‚0 disk.rbd2.out‚0! disk.nvme0n1.out‚ 1818.0234375! proc.memory.rssMB‚ 2932.140625 disk.rbd0.in‚0 disk.sdb.out‚59.94970703125 disk.sda2.in‚ 0.27734375 cpu‚24.602991859542257Îþ+]æ‚(:… „›çÔž©Â‡ gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚97.974' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚24.4935 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì¢\Iç‚(:¾ „›çÔ ½˜ˆ% disk./.usageGB‚2974.3137702941895 disk.md0p1.in‚ 0.4296875 disk.sdb.in‚ 0.15234375 cpu‚24.613736556845375 disk.rbd0.out‚0 disk.rbd2.in‚0" disk.nvme1n1.out‚ 5904.20703125 disk.dm-0.in‚ 361.48046875 disk.md0p1.out‚ 59.85546875 disk.rbd0.in‚0 disk.sda.out‚61.81396484375$ memory_percent‚36.83784008026123 disk.sdb.out‚61.81396484375 disk.sdb1.out‚0 disk.sdb2.in‚ 0.15234375 proc.cpu.threads‚17 network.recv‚273222 disk.nvme0n1.in‚ 283.85546875 disk.sda1.in‚0 disk.sdb1.in‚0 disk.sda.in‚ 0.27734375* proc.memory.availableMB‚10348.48828125 disk.dm-0.out‚ 7780.921875! proc.memory.rssMB‚ 2932.140625 network.sent‚713246) disk./.usagePercent‚83.16305786444794 disk.sda2.out‚61.81396484375 disk.sdb2.out‚61.81396484375" disk.nvme0n1.out‚ 1876.71484375 disk.sda1.out‚0 disk.sda2.in‚ 0.27734375 disk.md0.in‚ 0.4296875 disk.md0.out‚ 59.85546875 disk.rbd2.out‚0 disk.nvme1n1.in‚77.625* proc.memory.percent‚17.896366119384766ý’MÆ8è‚(:­ “›çÔ©ý“‰ cpu‚24.62383464427204 disk.sdb2.in‚ 0.15234375 disk.md0.out‚ 61.30078125 disk.sdb.in‚ 0.15234375 disk.sdb1.out‚0 disk.sda.out‚ 63.310546875 disk.sdb2.out‚ 63.310546875 disk.md0.in‚ 0.4296875 disk.rbd0.out‚0 disk.nvme0n1.out‚ 1938.03125 disk.sdb1.in‚0 disk.sda1.out‚0 disk.sda2.in‚ 0.27734375* proc.memory.percent‚17.896366119384766) disk./.usagePercent‚83.16305946435592 disk.nvme1n1.in‚77.625 disk.sdb.out‚ 63.310546875 disk.sda.in‚ 0.27734375 disk.sda1.in‚0 proc.cpu.threads‚17% disk./.usageGB‚2974.3138275146484 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.nvme1n1.out‚ 5904.40625 disk.md0p1.in‚ 0.4296875* proc.memory.availableMB‚10348.73828125 network.sent‚716247 disk.nvme0n1.in‚ 283.85546875 disk.dm-0.out‚ 7842.4375! proc.memory.rssMB‚ 2932.140625 network.recv‚277064 disk.dm-0.in‚ 361.48046875 disk.md0p1.out‚ 61.30078125 disk.rbd2.out‚0 disk.sda2.out‚ 63.310546875$ memory_percent‚36.83631420135498(Xšo’é‚(:‡ “›çÔ³‰…š gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚105.443' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚26.36075 gpu.0.smClock‚1230 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.”,0ê+ _wandbruntime‚593  _runtime‚593*ñŸ[Ãë½ & lr-AdamW/pg1‚0.0003402827943505082 & lr-AdamW/pg2‚0.0003402827943505082 trainer/global_step‚499 $ _timestamp‚1.788464540661613e+09 _runtime‚ 593.846729394 _step‚36Én¦ÇìÁ & lr-AdamW/pg2‚0.0003402827943505082 trainer/global_step‚499 $ _timestamp‚1.788464540661613e+09 _runtime‚ 593.846729394 _step‚36 & lr-AdamW/pg1‚0.0003402827943505082$mŠé0í+ _wandbruntime‚593  _runtime‚593S/dm î­O IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20641837371904 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 981317971456 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚20859084485120 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 trainer/global_step‚499 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚676 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚20642088276480 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚21859066513920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8894508 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 276706304 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 999681382400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/¼z€ÏFoversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 999958088704 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9149989 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9149989 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚19757077475708 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚18778767653840 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚20642088276480 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13392914 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 978538795404 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13392914 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚20641837371904 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3485464 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚18778784693200 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 978309821868 _runtime‚ 593.848370982 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 _step‚37 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9149991 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚19660770305024 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9149991 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚19660770305024 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22542905 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5409718 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5409720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚19660752302080 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚21858765867520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22518057 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚19757323488604 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 981317971456 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 981085069824 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3484790 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚19660752302080 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 % _timestamp‚1.7884645406627953e+09X ›ƒ·'ï±O XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚20642088276480 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚21859066513920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8894508 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 276706304 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 999681382400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 999958088704 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9149989 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9149989 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚19757077475708 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚18778767653840 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚20642088276480 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13392914 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 978538795404 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13392914 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚20641837371904 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3485464 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚18778784693200 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 978309821868 _runtime‚ 593.848370982 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 _step‚37 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9149991 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚19660770305024 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9149991 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚19660770305024 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22542905 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5409718 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5409720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚19660752302080 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚21858765867520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22518057 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚19757323488604 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 981317971456 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 981085069824 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3484790 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚19660752302080 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 % _timestamp‚1.7884645406627953e+09 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20641837371904 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 981317971456 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚20859084485120 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 trainer/global_step‚499 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚676 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582%ŠØ3ö0ð+ _wandbruntime‚594  _runtime‚594ßV¨ë¾&ñ¸M A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13394760 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚21902446378496 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22563024 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8912476 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8911820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3491300 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13419632 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚18816315050624 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282984448 trainer/global_step‚499 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚20683087434752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22587898 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚654 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9168266 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _step‚38 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚19700056342016 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 983049095680 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5420522 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚19796566322118 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚19700038339072 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚20900768455168 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20683338352640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚19796812335222 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚18816298011264 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13394760 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 983282010624 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3491954 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 983282010624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306924544 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 $ _timestamp‚1.788464541680022e+09 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚20683338352640 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚19700056342016 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1001677923328 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚ 594.865620008 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22587898 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22563024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚21902753303040 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9168264 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 980497284598 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13419632 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608K¼–ðòò¼M [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 980497284598 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13419632 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13394760 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚21902446378496 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22563024 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8912476 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8911820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3491300 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13419632 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚18816315050624 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282984448 trainer/global_step‚499 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚20683087434752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22587898 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚654 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9168266 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _step‚38 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚19700056342016 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 983049095680 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5420522 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚19796566322118 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚19700038339072 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚20900768455168 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20683338352640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚19796812335222 YFDeviceStatsMonitor.œ.”ÞÐon_train_batch_end/requested_bytes.large_pool.freed‚18816298011264 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13394760 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 983282010624 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3491954 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 983282010624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306924544 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 $ _timestamp‚1.788464541680022e+09 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚20683338352640 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚19700056342016 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1001677923328 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚ 594.865620008 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22587898 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22563024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚21902753303040 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9168264&%ŸM0ó+ _wandbruntime‚594  _runtime‚594PŠ… RôÌ 0loss/FBt4/assignment_loss‚0.4706514525110833 0loss/SRqqt2/detection_loss‚4.367302170821598 1loss/SRqqt4/detection_loss‚4.6681673767474985 .loss/FRt3/assignment_loss‚5.45566397382502 /loss/FBt1/detection_loss‚2.2033763899220675 .loss/FBt3/detection_loss‚4.030255963060039 _step‚39 0loss/FRt1/assignment_loss‚6.6454814821481705 /loss/FRt4/assignment_loss‚6.964917093515396 1loss/SRqqt2/assignment_loss‚3.249644194330488 2loss/SRqqt4/assignment_loss‚3.0652307913853574 0loss/FBt1/assignment_loss‚0.5911099242441803 /loss/FRt2/detection_loss‚2.5823915719985964 trainer/global_step‚499 % _timestamp‚1.7884645416818686e+09 1loss/FBt2/assignment_loss‚0.37490863440325484 epoch‚0 .loss/FRt1/detection_loss‚2.349493194837123 .loss/FRt4/detection_loss‚4.022316152229905 2loss/SRqqt3/assignment_loss‚2.4837687332908827 1loss/FBt3/assignment_loss‚0.18854775292210332 1loss/SRqqt1/detection_loss‚1.0108818071840489 1loss/SRqqt3/detection_loss‚1.5537407968628716 .loss/FBt2/detection_loss‚7.605098068714142 .loss/FBt4/detection_loss‚4.882901777029037 %loss/total_loss‚80.99660884309807 /loss/FRt2/assignment_loss‚6.715272092819214 2loss/SRqqt1/assignment_loss‚2.9623572900891304 .loss/FRt3/detection_loss‚2.553130158206873 _runtime‚ 594.866588023bðÙVõÐ .loss/FRt1/detection_loss‚2.349493194837123 .loss/FRt4/detection_loss‚4.022316152229905 2loss/SRqqt3/assignment_loss‚2.4837687332908827 1loss/FBt3/assignment_loss‚0.18854775292210332 1loss/SRqqt1/detection_loss‚1.0108818071840489 1loss/SRqqt3/detection_loss‚1.5537407968628716 .loss/FBt2/detection_loss‚7.605098068714142 .loss/FBt4/detection_loss‚4.882901777029037 %loss/total_loss‚80.99660884309807 /loss/FRt2/assignment_loss‚6.715272092819214 2loss/SRqqt1/assignment_loss‚2.9623572900891304 .loss/FRt3/detection_loss‚2.553130158206873 _runtime‚ 594.866588023 0loss/FBt4/assignment_loss‚0.4706514525110833 0loss/SRqqt2/detection_loss‚4.367302170821598 1loss/SRqqt4/detection_loss‚4.6681673767474985 .loss/FRt3/assignment_loss‚5.45566397382502 /loss/FBt1/detection_loss‚2.2033763899220675 .loss/FBt3/detection_loss‚4.030255963060039 _step‚39 0loss/FRt1/assignment_loss‚6.6454814821481705 /loss/FRt4/assignment_loss‚6.964917093515396 1loss/SRqqt2/assignment_loss‚3.249644194330488 2loss/SRqqt4/assignment_loss‚3.0652307913853574 0loss/FBt1/assignment_loss‚0.5911099242441803 /loss/FRt2/detection_loss‚2.5823915719985964 trainer/global_step‚499 % _timestamp‚1.7884645416818686e+09 1loss/FBt2/assignment_loss‚0.37490863440325484 epoch‚0'’+Ñ”‘ö‚(:† ¢›çÔØóʇ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚101.986' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.4965 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0s¥ÚJ÷‚(:¿ ¢›çÔì›»ˆ disk.dm-0.out‚ 8063.52734375 disk.sda2.out‚64.67431640625 disk.md0.out‚ 62.62890625 disk.sdb.out‚64.67431640625 disk.sda.out‚64.67431640625! disk.nvme0n1.out‚ 2026.0078125 disk.sda1.out‚0% memory_percent‚36.836791038513184 network.sent‚735558" disk.nvme1n1.out‚ 6037.51953125 disk.sdb1.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sdb2.in‚ 0.15234375 disk.sdb2.out‚64.67431640625 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.nvme1n1.in‚77.625 disk.sdb.in‚ 0.15234375) disk./.usagePercent‚83.14823503697792! proc.memory.rssMB‚ 2932.140625$ disk./.usageGB‚2973.783634185791 disk.sda2.in‚ 0.27734375 disk.rbd2.out‚0 disk.sdb1.out‚0 disk.dm-0.in‚ 361.48046875* proc.memory.percent‚17.896366119384766 disk.md0p1.out‚ 62.62890625* proc.memory.availableMB‚10348.66015625 network.recv‚281968 disk.md0p1.in‚ 0.4296875 disk.md0.in‚ 0.4296875 disk.rbd0.in‚0 cpu‚24.633194191404655 disk.sda.in‚ 0.27734375 disk.nvme0n1.in‚ 283.85546875ý1dË›ø‚(: ±›çÔØèɇ gpu.0.gpu‚8.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.455' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚27.363749999999996 gpu.0.smClock‚1230 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CÞEù‚(:º ±›çÔÊÑŸˆ disk.sdb.in‚ 0.15234375% disk./.usageGB‚2973.7833862304688 disk.sdb2.in‚ 0.15234375 disk.rbd0.out‚0* proc.memory.percent‚17.896366119384766) disk./.usagePercent‚83.14822810404334 disk.nvme1n1.out‚ 6237.421875 disk.sdb.out‚66.19384765625 disk.dm-0.in‚ 361.48046875 disk.md0p1.in‚ 0.4296875 disk.rbd2.out‚0 network.sent‚774019 network.recv‚287216 disk.rbd2.in‚0 disk.nvme0n1.in‚ 283.85546875 disk.md0.in‚ 0.4296875 disk.sda1.in‚0 disk.sda2.out‚66.19384765625( proc.memory.availableMB‚ 10348.796875 disk.sda2.in‚ 0.27734375 disk.sdb2.out‚66.19384765625 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sda.out‚66.19384765625! disk.nvme0n1.out‚ 2075.8828125 disk.md0p1.out‚ 64.09765625 cpu‚24.642082832681087 disk.nvme1n1.in‚77.625 disk.sdb1.out‚0$ memory_percent‚36.83595657348633 proc.cpu.threads‚17 disk.sda.in‚ 0.27734375 disk.dm-0.out‚ 8313.3046875 disk.sda1.out‚0! proc.memory.rssMB‚ 2932.140625 disk.md0.out‚ 64.09765625…²s¿ú‚(:„ À›çÔÑöÓ‡ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚86.834' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚21.7085 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šºÄ©;û‚(:° À›çÔí݉ˆ disk.sda2.out‚ 68.4462890625 proc.cpu.threads‚17% disk./.usageGB‚2973.7326850891113 disk.sdb2.in‚ 0.15234375 disk.sda1.out‚0 disk.md0p1.out‚ 66.296875 network.sent‚780161 network.recv‚291570 disk.rbd0.out‚0 disk.nvme1n1.out‚ 6393.53125 disk.sda.in‚ 0.27734375 disk.md0p1.in‚ 0.4296875$ memory_percent‚36.83595657348633( proc.memory.availableMB‚ 10348.796875) disk./.usagePercent‚83.14681047891091 disk.sdb2.out‚ 68.4462890625 disk.rbd2.out‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.nvme0n1.in‚ 283.85546875! disk.nvme0n1.out‚ 2076.5390625 disk.dm-0.in‚ 361.48046875 disk.sda2.in‚ 0.27734375 disk.sdb.in‚ 0.15234375 disk.md0.in‚ 0.4296875 disk.nvme1n1.in‚77.625 disk.sdb.out‚ 68.4462890625 disk.sdb1.in‚0 disk.dm-0.out‚ 8470.0703125! proc.memory.rssMB‚ 2932.140625* proc.memory.percent‚17.896366119384766 cpu‚24.65056622621744 disk.md0.out‚ 66.296875 disk.sda.out‚ 68.4462890625 disk.sda1.in‚0Éç6Æ‘ü‚(:† Ï›çÔÌŸ¢‡ gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚93.631' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚23.40775 gpu.0.smClock‚1215 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zÔ„U;ý‚(:° Ï›çÔÏíˆ disk.sdb1.out‚0 disk.sda1.in‚0 disk.sdb2.out‚ 70.138671875 disk.md0.out‚ 67.94140625* proc.memory.percent‚17.896366119384766 disk.md0p1.out‚ 67.94140625 disk.sdb2.in‚ 0.15234375 disk.nvme1n1.out‚ 6570.0625( proc.memory.availableMB‚ 10348.796875 disk.sda.in‚ 0.27734375 disk.sda.out‚ 70.138671875" disk.nvme0n1.out‚ 2157.34765625 disk.sda2.out‚ 70.138671875 disk.rbd2.in‚0 disk.sdb.in‚ 0.15234375 network.sent‚783503 disk.nvme0n1.in‚ 283.85546875 disk.dm-0.in‚ 361.48046875 disk.rbd2.out‚0 disk.sdb.out‚ 70.138671875% disk./.usageGB‚2973.8115158081055 disk.sda1.out‚0 disk.nvme1n1.in‚77.625 network.recv‚295571( disk./.usagePercent‚83.1490146188072 disk.sdb1.in‚0 disk.dm-0.out‚ 8727.41015625 disk.md0p1.in‚ 0.4296875 disk.md0.in‚ 0.4296875 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚36.83595657348633! proc.memory.rssMB‚ 2932.140625 cpu‚24.65903495032958 disk.sda2.in‚ 0.27734375 proc.cpu.threads‚17W…Eï0þ+ _wandbruntime‚651  _runtime‚651b1jÅÿ¿ ' lr-AdamW/pg1‚0.00037437926673031865 ' lr-AdamW/pg2‚0.00037437926673031865 trainer/global_step‚549 % _timestamp‚1.7884645986212525e+09 _runtime‚ 651.80641496 _step‚40œƒê—ɀà _runtime‚ 651.80641496 _step‚40 ' lr-AdamW/pg1‚0.00037437926673031865 ' lr-AdamW/pg2‚0.00037437926673031865 trainer/global_step‚549 % _timestamp‚1.7884645986212525e+09(T„”0+ _wandbruntime‚651  _runtime‚651‡Çm¿'‚¹O QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22704354928128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10063739 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1099694685184 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚703 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _runtime‚ 651.807987677 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚21731779400312 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚20655302564400 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1079300776448 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚22704605832704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7884645986224291e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚24042977672704 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10063741 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1079533678080 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1079300776448 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚22943282987520 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚21731533387416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1076476835912 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9780130 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24794607 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1099971391488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚22943306927616 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10063739 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14730866 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1079533678080 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3830312 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582HüË€Ü!ƒ½O @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1079300776448 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚22943282987520 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚21731533387416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1076476835912 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9780130 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24794607 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1099971391488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚22943306927616 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10063739 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14730866 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1079533678080 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3830312 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22704354928128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10063739 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1099694685184 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚703 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _runtime‚ 651.807987677 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitorzÉrç.on_train_batch_start/requested_bytes.all.allocated‚21731779400312 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚20655302564400 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1079300776448 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚22704605832704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7884645986224291e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚24042977672704 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10063741 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1079533678080 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920)÷ˆ7¹0„+ _wandbruntime‚652  _runtime‚652Çë#É&…ÃM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10082014 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5960620 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14732712 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚21664340188672 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1081497717248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1081264802304 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚20692832921824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9797590 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚21664340188672 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚21664358191616 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚24086966894080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚21771022233826 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1081264802304 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22745855908864 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24814726 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 287178752 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10082014 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24814726 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚21664358191616 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚20692815882464 _runtime‚ 652.996909606 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24839600 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚710 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _step‚42 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚22984990897664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14757584 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10082016 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚22745604990976 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1101688817664 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1078435325106 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1078206351362 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1081497717248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10082016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14732712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 % _timestamp‚1.7884645998097456e+09 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚21771268246930 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚22984966957568 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14757584 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9798302 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 311118848 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1101975996416 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0m¹|JÍ&†ÇM _step‚42 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚22984990897664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14757584 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10082016 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚22745604990976 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1101688817664 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1078435325106 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1078206351362 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1081497717248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10082016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14732712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 % _timestamp‚1.7884645998097456e+09 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚21771268246930 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚22984966957568 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14757584 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9798302 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 311118848 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1101975996416 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10082014 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5960620 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14732712 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚21664340188672 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1081497717248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1081264802304 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚20692832921824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9797590 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚21664340188672 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚21664358191616 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚24086966894080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚21771022233826 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1081264802304 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22745855908864 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24814726 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 287178752 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10082014 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24814726 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚21664358191616 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚20692815882464 _runtime‚ 652.996909606 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24839600 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚710 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309*X•¼0‡+ _wandbruntime‚652  _runtime‚652÷\*lNˆÈ 2loss/SRqqt3/assignment_loss‚2.7342040974360247 1loss/SRqqt3/detection_loss‚1.0998003234943519 -loss/FRt1/detection_loss‚2.76474968791008 /loss/FRt2/detection_loss‚2.9395970851182938 -loss/FBt1/detection_loss‚6.23724253168214 0loss/FRt1/assignment_loss‚6.5997608947753905 /loss/FRt4/assignment_loss‚6.652800912442415 1loss/SRqqt4/assignment_loss‚2.632665411118538 0loss/FBt1/assignment_loss‚0.8594983882374234 epoch‚0 trainer/global_step‚549 % _timestamp‚1.7884645998133857e+09 2loss/SRqqt2/assignment_loss‚3.4269385867648654 0loss/FBt2/assignment_loss‚0.6171330187547331 0loss/FBt4/assignment_loss‚0.4240557377559266 .loss/FRt4/detection_loss‚4.429994711409444 0loss/SRqqt1/detection_loss‚4.292008160668261 $loss/total_loss‚81.6958899186535 /loss/FRt2/assignment_loss‚6.455874562263489 0loss/FBt3/assignment_loss‚0.2822580611368342 /loss/FRt3/detection_loss‚1.8938063697020213 1loss/SRqqt4/detection_loss‚3.4757010667554793 .loss/FBt2/detection_loss‚7.898390895790524 _runtime‚ 652.998363762 /loss/FRt3/assignment_loss‚4.830601743062338 0loss/SRqqt2/detection_loss‚3.104457492629687 _step‚43 /loss/FBt4/detection_loss‚3.2130320046629226 /loss/FBt3/detection_loss‚1.8084239784317713 1loss/SRqqt1/assignment_loss‚3.022894196650561°öÄ—R‰Ì 2loss/SRqqt2/assignment_loss‚3.4269385867648654 0loss/FBt2/assignment_loss‚0.6171330187547331 0loss/FBt4/assignment_loss‚0.4240557377559266 .loss/FRt4/detection_loss‚4.429994711409444 0loss/SRqqt1/detection_loss‚4.292008160668261 $loss/total_loss‚81.6958899186535 /loss/FRt2/assignment_loss‚6.455874562263489 0loss/FBt3/assignment_loss‚0.2822580611368342 /loss/FRt3/detection_loss‚1.8938063697020213 1loss/SRqqt4/detection_loss‚3.4757010667554793 .loss/FBt2/detection_loss‚7.898390895790524 _runtime‚ 652.998363762 /loss/FRt3/assignment_loss‚4.830601743062338 0loss/SRqqt2/detection_loss‚3.104457492629687 _step‚43 /loss/FBt4/detection_loss‚3.2130320046629226 /loss/FBt3/detection_loss‚1.8084239784317713 1loss/SRqqt1/assignment_loss‚3.022894196650561 2loss/SRqqt3/assignment_loss‚2.7342040974360247 1loss/SRqqt3/detection_loss‚1.0998003234943519 -loss/FRt1/detection_loss‚2.76474968791008 /loss/FRt2/detection_loss‚2.9395970851182938 -loss/FBt1/detection_loss‚6.23724253168214 0loss/FRt1/assignment_loss‚6.5997608947753905 /loss/FRt4/assignment_loss‚6.652800912442415 1loss/SRqqt4/assignment_loss‚2.632665411118538 0loss/FBt1/assignment_loss‚0.8594983882374234 epoch‚0 trainer/global_step‚549 % _timestamp‚1.7884645998133857e+09+¶›ÈŠ‚(:„ Þ›çÔŽ¨Ï‡ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚92.404' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚23.101 gpu.0.smClock‚1170 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/˜®?=‹‚(:² Þ›çÔ¬Õˆ disk.sdb.in‚ 0.15234375 disk.sdb.out‚ 71.41796875 network.sent‚789834 disk.sdb1.out‚0 disk.sda.in‚ 0.27734375 disk.sda.out‚ 71.41796875 disk.sdb2.out‚ 71.41796875* proc.memory.availableMB‚10348.75390625 disk.md0p1.out‚ 69.17578125 disk.rbd0.in‚0 disk.nvme1n1.in‚77.625 disk.dm-0.out‚ 8892.96484375 disk.sda1.out‚0 disk.sdb2.in‚ 0.15234375 proc.cpu.threads‚17 disk.nvme1n1.out‚ 6703.140625 disk.rbd2.out‚0 disk.nvme0n1.in‚ 283.85546875$ memory_percent‚36.83621883392334! proc.memory.rssMB‚ 2932.140625) disk./.usagePercent‚83.14995547136105" disk.nvme0n1.out‚ 2189.82421875 disk.dm-0.in‚ 361.48046875 disk.md0p1.in‚ 0.4296875 disk.md0.in‚ 0.4296875 disk.sda1.in‚0 disk.sda2.out‚ 71.41796875 network.recv‚300119% disk./.usageGB‚2973.8451652526855 disk.md0.out‚ 69.17578125 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda2.in‚ 0.27734375* proc.memory.percent‚17.896366119384766 disk.sdb1.in‚0 cpu‚24.666723065245712¸p2‘Œ‚(:† í›çÔšè߇ gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚93.631' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚23.40775 gpu.0.smClock‚1230 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0GÆ*¼\‚(:Ñ í›çÔ«”ƈ* proc.memory.percent‚17.896366119384766 network.recv‚304922 cpu‚24.674494974352935 disk.nvme0n1.in‚ 283.85546875 disk.sdb.out‚73.42431640625 proc.cpu.threads‚17 network.sent‚840802 disk.sda2.in‚ 0.27734375 disk.md0.in‚ 0.4296875! disk.nvme1n1.out‚ 6836.3828125) proc.memory.availableMB‚ 10348.7109375% disk./.usageGB‚2973.9389152526855 disk.md0p1.out‚ 71.1171875 disk.sdb2.in‚ 0.15234375 disk.sdb1.in‚0 disk.sda.out‚73.42431640625 disk.sda1.in‚0 disk.sda2.out‚73.42431640625 disk.sdb.in‚ 0.15234375 disk.sda.in‚ 0.27734375 disk.sdb2.out‚73.42431640625 disk.nvme1n1.in‚77.625! proc.memory.rssMB‚ 2932.140625 disk.dm-0.out‚ 9085.2265625 disk.md0p1.in‚ 0.4296875 disk.md0.out‚ 71.1171875 disk.rbd0.out‚ 0.07421875 disk.rbd2.in‚0$ memory_percent‚36.83648109436035) disk./.usagePercent‚83.15257676059848 disk.nvme0n1.out‚ 2248.84375 disk.dm-0.in‚ 361.48046875 disk.sda1.out‚0 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 0.01171875 disk.sdb1.out‚0‚ÝAŽ‚(:… ü›çÔºÀÚ‡ gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚135.184' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.796 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ËË›øQ‚(:Æ ü›çÔ«¬›ˆ) proc.memory.availableMB‚ 10348.7109375) disk./.usagePercent‚83.15607181291503 disk.sda1.out‚0 disk.sdb2.in‚ 0.15234375 disk.rbd0.out‚ 0.07421875 disk.rbd2.in‚0 disk.sda.out‚ 74.88671875 network.sent‚847188 network.recv‚309718 disk.dm-0.out‚ 9335.546875 disk.md0p1.out‚ 72.546875 disk.rbd0.in‚0.015625 disk.nvme1n1.in‚77.625 disk.sdb1.in‚0 disk.dm-0.in‚ 361.48046875 disk.sda1.in‚0 disk.sda2.out‚ 74.88671875 disk.sdb2.out‚ 74.88671875 disk.md0.in‚ 0.4296875 disk.sdb.in‚ 0.15234375 disk.sdb.out‚ 74.88671875 proc.cpu.threads‚17 disk.md0p1.in‚ 0.4296875" disk.nvme1n1.out‚ 6969.44921875$ memory_percent‚36.83648109436035! proc.memory.rssMB‚ 2932.140625% disk./.usageGB‚2974.0639152526855 disk.sda2.in‚ 0.27734375 disk.md0.out‚ 72.546875 disk.rbd2.out‚ 0.07421875 disk.sdb1.out‚0" disk.nvme0n1.out‚ 2366.09765625 disk.sda.in‚ 0.27734375* proc.memory.percent‚17.896366119384766 cpu‚24.681959759375207 disk.nvme0n1.in‚ 283.85546875ÕàvÖ’‚(:‡ ‹œçÔÂʇ gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚131.173' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.79325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ϧýXY‘‚(:Î ‹œçÔóîˆ disk.sda.in‚ 0.27734375 disk.nvme0n1.in‚ 293.17578125 disk.sda2.out‚76.43994140625 disk.md0p1.in‚ 0.4296875 disk.sdb2.in‚ 0.15234375 disk.rbd2.out‚ 0.07421875 disk.nvme1n1.out‚ 6970.03125$ memory_percent‚36.83810234069824 disk.sdb.out‚76.43994140625 disk.sdb2.out‚76.43994140625 disk.md0.in‚ 0.4296875 disk.md0.out‚ 74.046875 disk.rbd0.out‚ 0.1533203125) proc.memory.availableMB‚ 10348.4453125* proc.memory.percent‚17.896366119384766 proc.cpu.threads‚17 network.recv‚313535) disk./.usagePercent‚83.16306319747456 disk.sdb1.out‚0 disk.rbd0.in‚0.015625! proc.memory.rssMB‚ 2932.140625 network.sent‚852414 disk.sdb.in‚ 0.15234375 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0p1.out‚ 74.046875 disk.sda2.in‚ 0.27734375 disk.nvme1n1.in‚77.625% disk./.usageGB‚2974.3139610290527 disk.sdb1.in‚0 disk.sda.out‚76.43994140625 disk.nvme0n1.out‚ 2431.734375 disk.dm-0.out‚ 9401.765625 disk.rbd2.in‚0 cpu‚24.68877280009847 disk.dm-0.in‚ 370.80078125Ô¼V0’+ _wandbruntime‚710  _runtime‚710CE–H“¼ _step‚44 & lr-AdamW/pg1‚0.0004084757391101291 & lr-AdamW/pg2‚0.0004084757391101291 trainer/global_step‚599 # _timestamp‚1.78846465725905e+09 _runtime‚ 710.444000613W¶ÐÆ”À & lr-AdamW/pg1‚0.0004084757391101291 & lr-AdamW/pg2‚0.0004084757391101291 trainer/global_step‚599 # _timestamp‚1.78846465725905e+09 _runtime‚ 710.444000613 _step‚44,®ŠHß0•+ _wandbruntime‚710  _runtime‚710ÿ›Š–»O =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10977491 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16043972 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚45 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1177749384704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚25027505430016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304840704 E?DeviceStatsMonitor.on_train_batch_start/oversize_segmentsæ¨îß7 .freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1177749384704 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27046309 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6489920 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1174185902884 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1177516483072 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24766872484352 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1177516483072 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚23589374004224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚23705989299124 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1199970643456 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚23589356001280 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4178908 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚689 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6489918 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚26227476073472 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16068818 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10669517 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 % _timestamp‚1.7884646572602339e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1177749384704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚25027505430016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304840704 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1177749384704 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27046309 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6489920 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1174185902884 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1177516483072 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24766872484352 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1177516483072 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚23589374004224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚23705989299124 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1199970643456 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚23589356001280 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4178908 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚689 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6489918 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚26227476073472 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16068818 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10669517 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 % _timestamp‚1.7884646572602339e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚22569333753664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10995764 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚23628642038272 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27066428 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _runtime‚ 711.402751273 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10686873 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚24808373465088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1176373365614 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 % _timestamp‚1.7884646582171335e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚23745478145534 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10995766 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚24808122547200 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚23628642038272 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10687586 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚25069165459968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27091302 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10995764 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16095536 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16070664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚711 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1201969972736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚26271159372800 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 trainer/global_step‚599 _step‚46 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚25069189400064 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16095536 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚23745724158638 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16070664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚713 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6500722 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1201682793984 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27066428 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4186153 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1179713423872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚26270848253952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1179713423872­úžæšÉM aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1201969972736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚26271159372800 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 trainer/global_step‚599 _step‚46 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚25069189400064 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16095536 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚23745724158638 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16070664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚713 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6500722 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1201682793984 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27066428 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4186153 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1179713423872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚26270848253952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1179713423872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10995766 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/r£ažéequested_bytes.large_pool.freed‚22569333753664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10995764 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚23628642038272 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27066428 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _runtime‚ 711.402751273 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10686873 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚24808373465088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1176373365614 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 % _timestamp‚1.7884646582171335e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚23745478145534 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10995766 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚24808122547200 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚23628642038272 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10687586 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚25069165459968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27091302 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10995764 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16095536 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16070664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚711 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480.ÕPà.0›+ _wandbruntime‚711  _runtime‚711„”ªôRœÌ trainer/global_step‚599 /loss/FRt1/assignment_loss‚6.639442025398721 1loss/FBt4/assignment_loss‚0.49024421293021686 .loss/FRt2/detection_loss‚3.642690827449163 epoch‚0 .loss/FRt2/assignment_loss‚6.13561487197876 1loss/SRqqt4/assignment_loss‚2.832833143075307 % _timestamp‚1.7884646582191722e+09 _step‚47 /loss/FRt3/detection_loss‚1.7603399766609074 .loss/FBt2/detection_loss‚6.381053379603794 /loss/FBt3/detection_loss‚1.6060700290012817 0loss/FBt2/assignment_loss‚0.8015991563526248 /loss/FRt3/assignment_loss‚4.117550015449524 1loss/FBt1/assignment_loss‚0.32400467448261383 /loss/FRt4/detection_loss‚3.9035645205041636 0loss/SRqqt3/detection_loss‚1.067735748425885 0loss/SRqqt4/detection_loss‚3.288205076257388 /loss/FBt4/detection_loss‚4.2403963680068655 _runtime‚ 711.403914098 2loss/SRqqt1/assignment_loss‚2.8716447276453816 .loss/FRt1/detection_loss‚2.843197579894747 1loss/SRqqt2/detection_loss‚3.9341947617738144 /loss/FBt1/detection_loss‚6.8456900435335495 1loss/FBt3/assignment_loss‚0.35033687261834817 0loss/SRqqt1/detection_loss‚4.063372577390363 %loss/total_loss‚80.18673684616908 /loss/FRt4/assignment_loss‚6.416012660316799 2loss/SRqqt2/assignment_loss‚3.2727723639944326 1loss/SRqqt3/assignment_loss‚2.358171233424434.Òè±VÐ /loss/FRt4/assignment_loss‚6.416012660316799 2loss/SRqqt2/assignment_loss‚3.2727723639944326 1loss/SRqqt3/assignment_loss‚2.358171233424434 trainer/global_step‚599 /loss/FRt1/assignment_loss‚6.639442025398721 1loss/FBt4/assignment_loss‚0.49024421293021686 .loss/FRt2/detection_loss‚3.642690827449163 epoch‚0 .loss/FRt2/assignment_loss‚6.13561487197876 1loss/SRqqt4/assignment_loss‚2.832833143075307 % _timestamp‚1.7884646582191722e+09 _step‚47 /loss/FRt3/detection_loss‚1.7603399766609074 .loss/FBt2/detection_loss‚6.381053379603794 /loss/FBt3/detection_loss‚1.6060700290012817 0loss/FBt2/assignment_loss‚0.8015991563526248 /loss/FRt3/assignment_loss‚4.117550015449524 1loss/FBt1/assignment_loss‚0.32400467448261383 /loss/FRt4/detection_loss‚3.9035645205041636 0loss/SRqqt3/detection_loss‚1.067735748425885 0loss/SRqqt4/detection_loss‚3.288205076257388 /loss/FBt4/detection_loss‚4.2403963680068655 _runtime‚ 711.403914098 2loss/SRqqt1/assignment_loss‚2.8716447276453816 .loss/FRt1/detection_loss‚2.843197579894747 1loss/SRqqt2/detection_loss‚3.9341947617738144 /loss/FBt1/detection_loss‚6.8456900435335495 1loss/FBt3/assignment_loss‚0.35033687261834817 0loss/SRqqt1/detection_loss‚4.063372577390363 %loss/total_loss‚80.18673684616908/_­c’ž‚(:‡ šœçÔž‘ˆ gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚138.919' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.72975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0IeƒhZŸ‚(:Ï šœçÔï몈 disk.dm-0.out‚ 9617.81640625 disk.sdb.in‚ 0.15234375 disk.sdb1.in‚0) proc.memory.availableMB‚ 10347.8828125! disk.nvme0n1.out‚ 2515.6953125 disk.sda1.out‚0 disk.rbd0.out‚ 0.1533203125! proc.memory.rssMB‚ 2932.140625 cpu‚24.69561133463812 network.recv‚318048 disk.md0p1.in‚ 0.4296875 disk.rbd0.in‚0.015625 disk.sda.out‚ 77.857421875 disk.sdb1.out‚0 network.sent‚869541 disk.rbd2.in‚0 disk.rbd2.out‚ 0.1533203125 disk.nvme1n1.in‚77.625% memory_percent‚36.841535568237305* proc.memory.percent‚17.896366119384766) disk./.usagePercent‚83.14331542659434 disk.dm-0.in‚ 370.94140625 disk.sda2.out‚ 77.857421875 disk.md0.in‚ 0.4296875 proc.cpu.threads‚17 disk.sda1.in‚0" disk.nvme1n1.out‚ 7103.12109375% disk./.usageGB‚2973.6076850891113 disk.sda.in‚ 0.27734375 disk.sda2.in‚ 0.27734375 disk.md0p1.out‚ 75.421875 disk.sdb2.in‚ 0.15234375 disk.sdb2.out‚ 77.857421875 disk.md0.out‚ 75.421875 disk.sdb.out‚ 77.857421875 disk.nvme0n1.in‚ 293.316406258ºv‘ ‚(:† ©œçÔΣӇ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚118.142' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.5355 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DˆRc¡‚(:Ø ©œçÔÈû„ˆ disk.dm-0.out‚ 9757.80859375 disk.sda1.in‚0 disk.sda2.in‚ 0.27734375 disk.rbd0.out‚ 0.21630859375 disk.md0p1.out‚ 77.41796875 disk.sdb2.in‚ 0.15234375 disk.md0.in‚ 0.4296875* proc.memory.percent‚17.896389961242676 disk.sda2.out‚ 79.9111328125 disk.sda.out‚ 79.9111328125 disk.nvme0n1.out‚ 2542.609375# proc.memory.rssMB‚ 2932.14453125 disk.sdb2.out‚ 79.9111328125 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 0.1533203125" disk.nvme1n1.out‚ 7215.19921875 disk.sdb1.in‚0 disk.sdb1.out‚0 cpu‚24.701899002447263 proc.cpu.threads‚17 disk.sda.in‚ 0.27734375 disk.md0p1.in‚ 0.4296875$ disk./.usageGB‚2973.607219696045 disk.dm-0.in‚ 370.94140625 disk.rbd2.in‚0 disk.sdb.in‚ 0.15234375 disk.sdb.out‚ 79.9111328125) proc.memory.availableMB‚ 10348.5859375) disk./.usagePercent‚83.14330241400943 disk.sda1.out‚0 disk.md0.out‚ 77.41796875 disk.nvme0n1.in‚ 293.31640625$ memory_percent‚36.83724403381348 disk.nvme1n1.in‚77.625 network.sent‚906730 network.recv‚322070 ³Á+‘¢‚(:† ¸œçÔÑÌɇ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚134.298' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.5745 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ɱÄþU£‚(:Ê ¸œçԊ cpu‚24.708227122631722) disk./.usagePercent‚83.14679938621556 disk.rbd2.out‚ 0.21630859375 disk.sda1.out‚0 disk.nvme1n1.in‚77.625 network.sent‚912695 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.nvme0n1.in‚ 293.31640625 disk.dm-0.out‚ 9991.59375 proc.cpu.threads‚17 disk.nvme1n1.out‚ 7331.734375 disk.sda1.in‚0 disk.sdb2.out‚ 81.568359375# proc.memory.rssMB‚ 2932.14453125 disk.md0.in‚ 0.4296875 disk.sdb.in‚ 0.15234375 disk.sdb.out‚ 81.568359375 disk.sda2.in‚ 0.27734375 disk.sdb2.in‚ 0.15234375' proc.memory.availableMB‚ 10348.59375 disk.rbd0.out‚ 0.21630859375 disk.nvme0n1.out‚ 2659.859375* proc.memory.percent‚17.896389961242676 disk.md0p1.out‚79.03125 network.recv‚326407 disk.sdb1.in‚0 disk.dm-0.in‚ 370.94140625 disk.sda2.out‚ 81.568359375% memory_percent‚36.837196350097656 disk.md0p1.in‚ 0.4296875% disk./.usageGB‚2973.7322883605957 disk.md0.out‚79.03125 disk.sdb1.out‚0 disk.sda.in‚ 0.27734375 disk.sda.out‚ 81.568359375 JZ{0¤+ _wandbruntime‚760  _runtime‚760ëõÉ´Æ¥À ' lr-AdamW/pg1‚0.00044257221148993946 ' lr-AdamW/pg2‚0.00044257221148993946 trainer/global_step‚649 % _timestamp‚1.7884647072416365e+09 _runtime‚ 760.426571805 _step‚48,-«!Ê¦Ä ' lr-AdamW/pg2‚0.00044257221148993946 trainer/global_step‚649 % _timestamp‚1.7884647072416365e+09 _runtime‚ 760.426571805 _step‚48 ' lr-AdamW/pg1‚0.000442572211489939460WÑÀ0§+ _wandbruntime‚760  _runtime‚760 ÏÒÅÀ'¨ºO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚26829390040576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚27111679992320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1300042775552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚25553675853824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚689 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7030020 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚25680445210832 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚26829390040576 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚26829640945152 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4528286 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚25553675853824 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29298011 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚27111703932416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1299747194880 % _timestamp‚1.7884647072426753e+09 _step‚49 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1275965091328 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚25680691223728 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚24408321267440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚649 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29273163 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚28411427187200 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚28411746707968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1272123943392 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1275732189696 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11558306 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1272352916928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚26829640945152 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 295580672 _runtime‚ 760.42770215 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4527597 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7030018 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11891241 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1275732189696 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17381924 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1275965091328 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17406770 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582îoÍBY#©¾O @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4527597 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7030018 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11891241 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1275732189696 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17381924 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1275965091328 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17406770 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚26829390040576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚27111679992320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1300042775552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚25553675853824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚689 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7030020 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚25680445210832 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚26829390040576 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚26829640945152 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4528286 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚25553675853824 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29298011 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚27111703932416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1299747194880 % _timestamp‚1.7884647072426753e+09 _step‚49 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1275965091328 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚25680691223728 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚24408321267440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚649 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29273163 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚28411427187200 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚28411746707968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1272123943392 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_hûÚAkpool.peak‚ 6561988608 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1275732189696 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11558306 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1272352916928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚26829640945152 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 295580672 _runtime‚ 760.42770215 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 71345111041ÛòŠ0ª+ _wandbruntime‚761  _runtime‚761œ:H¹Ê&«ÄM UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29343004 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17408616 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚25592961890816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚26870891021312 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11575074 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7040820 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚26870891021312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚25592943887872 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚50 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280887296 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚24445868664224 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29318130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4534933 $ _timestamp‚1.788464708189703e+09 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚27153363962368 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17433488 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4534254 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚25592961890816 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚27153387902464 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚25719934057242 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11909514 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11575755 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚25720180070346 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1274311406122 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1277929130496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1277696215552 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚26870640103424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11909514 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚28455112837120 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚26870891021312 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11575074 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7040820 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚26870891021312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚25592943887872 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚50 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280887296 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚24445868664224 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29318130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4534933 $ _timestamp‚1.788464708189703e+09 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚27153363962368 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17433488 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4534254 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚25592961890816 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚27153387902464 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚25719934057242 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11909514 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11575755 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚25720180070346 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1274311406122 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1277929130496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1277696215552 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚26870640103424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11909514 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚28455112837120 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29343004 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17408616 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚25592961890816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚27517977703424 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 F@DeviceStatsMonitor.on_train_batch_start/reserved_byœ@P? tes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12804989 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚27655147135436 # _timestamp‚1.78846476293813e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18744722 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚27654901122540 _runtime‚ 816.123484233 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18719876 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1370290957436 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚27517977703424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 287192064 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚706 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18719876 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1374180797952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1373947896320 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31524865 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18744722 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚28891907596800 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1370061983900 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚26284856178000 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 _step‚53 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚28892158501376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚29195878494720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1374180797952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28891907596800 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚30595852049920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚26284839138640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚27517959700480 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚28892158501376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12804989 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1373947896320 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1399662423040 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104ƵÃ'½½O `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚29195878494720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1374180797952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28891907596800 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚30595852049920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚26284839138640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚27517959700480 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚28892158501376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12804989 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1373947896320 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1399662423040 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚27517977703424 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12804989 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚27655147135436 # _timestamp‚1.78846476293813e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18744722 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚27654901122540 _runtime‚ 816.123484233 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18719876 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1370290957436 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚27517977703424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 287192064 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚706 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18719876 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1374180797952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1373947896320 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31524865 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18744722 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚28891907596800 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1370061983900 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚26284856178000 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 _step‚53 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚28892158501376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚365zD±H0¾+ _wandbruntime‚817  _runtime‚817ƒ¶CQÊ&¿ÄM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 319507456 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1375911922176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 295567360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 _runtime‚ 817.312742216 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚27557245737472 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18746568 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1401948026880 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31569832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚27557263740416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚26322386535424 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚54 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7580920 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12465992 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚30639214924288 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1376144837120 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 $ _timestamp‚1.788464764126213e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚704 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚28933408577536 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚27694389968950 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31569832 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚27557263740416 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18771440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚30639534431744 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚699 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12823266 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18746568 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚706 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12465286 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚29237586404864 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚27694635982054 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31594706 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7580922 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1372020472886 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18771440 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚27557245737472 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4885070 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4884366 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1401652459520 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚27557263740416 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18771440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚30639534431744 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚699 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12823266 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18746568 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚706 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12465286 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚29237586404864 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚27694635982054 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31594706 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7580922 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1372020472886 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18771440 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚27557245737472 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4885070 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4884366 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1401652459520 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1375911922176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 295567360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 _runtime‚ 817.312742216 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚27557245737472 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18746568 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1401948026880 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31569832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚27557263740416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚26322386535424 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚54 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7580920 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12465992 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚30639214924288 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1376144837120 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 $ _timestamp‚1.788464764126213e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚704 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚289334085775366Áz}—0Á+ _wandbruntime‚817  _runtime‚817Ù¡UÆOÂÉ 0loss/FRt3/assignment_loss‚3.7841746027086987 1loss/SRqqt2/assignment_loss‚2.958396037419637 1loss/FBt3/assignment_loss‚0.23061817276237662 .loss/FBt2/detection_loss‚8.521395921707153 1loss/SRqqt1/assignment_loss‚3.015310792361989 2loss/SRqqt3/assignment_loss‚2.5291786717957465 1loss/FBt2/assignment_loss‚0.42769603431224823 /loss/FRt2/detection_loss‚2.6887735180232837 -loss/SRqqt4/detection_loss‚2.901679397657 trainer/global_step‚699 _step‚55 /loss/FRt1/assignment_loss‚6.571316275485726 /loss/FRt2/assignment_loss‚6.346533028975777 0loss/FBt1/assignment_loss‚0.9048742949962616 0loss/SRqqt1/detection_loss‚2.597103691276382 _runtime‚ 817.314186233 .loss/FBt1/detection_loss‚19.35685294866562 %loss/total_loss‚88.32945779187231 1loss/SRqqt4/assignment_loss‚2.938478975460447 /loss/FRt3/detection_loss‚1.8719266684334954 .loss/FRt4/detection_loss‚2.990600710914981 1loss/SRqqt3/detection_loss‚0.7851396763600923 1loss/SRqqt2/detection_loss‚2.7159752303903755 0loss/FBt4/assignment_loss‚0.4137362839348498 /loss/FRt1/detection_loss‚2.6050537471161332 0loss/FRt4/assignment_loss‚6.2347011873798985 epoch‚0 /loss/FBt3/detection_loss‚1.1451153058971886 /loss/FBt4/detection_loss‚3.7948266178369523 # _timestamp‚1.78846476412934e+09´*4SÃÍ 0loss/FBt1/assignment_loss‚0.9048742949962616 0loss/SRqqt1/detection_loss‚2.597103691276382 _runtime‚ 817.314186233 .loss/FBt1/detection_loss‚19.35685294866562 %loss/total_loss‚88.32945779187231 1loss/SRqqt4/assignment_loss‚2.938478975460447 /loss/FRt3/detection_loss‚1.8719266684334954 .loss/FRt4/detection_loss‚2.990600710914981 1loss/SRqqt3/detection_loss‚0.7851396763600923 1loss/SRqqt2/detection_loss‚2.7159752303903755 0loss/FBt4/assignment_loss‚0.4137362839348498 /loss/FRt1/detection_loss‚2.6050537471161332 0loss/FRt4/assignment_loss‚6.2347011873798985 epoch‚0 /loss/FBt3/detection_loss‚1.1451153058971886 /loss/FBt4/detection_loss‚3.7948266178369523 # _timestamp‚1.78846476412934e+09 0loss/FRt3/assignment_loss‚3.7841746027086987 1loss/SRqqt2/assignment_loss‚2.958396037419637 1loss/FBt3/assignment_loss‚0.23061817276237662 .loss/FBt2/detection_loss‚8.521395921707153 1loss/SRqqt1/assignment_loss‚3.015310792361989 2loss/SRqqt3/assignment_loss‚2.5291786717957465 1loss/FBt2/assignment_loss‚0.42769603431224823 /loss/FRt2/detection_loss‚2.6887735180232837 -loss/SRqqt4/detection_loss‚2.901679397657 trainer/global_step‚699 _step‚55 /loss/FRt1/assignment_loss‚6.571316275485726 /loss/FRt2/assignment_loss‚6.3465330289757777%£€Q’Ä‚(:‡ ƒçÔø”ˆ gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚112.849' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.21225 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çÒœŠdÅ‚(:Ù ƒçÔ´®éˆ disk.rbd2.in‚0! disk.nvme1n1.out‚ 7334.5078125 disk.sdb1.in‚0* proc.memory.percent‚17.896389961242676 disk.nvme0n1.in‚ 293.31640625 disk.sda2.out‚ 89.2041015625 disk.rbd2.out‚ 0.279296875 disk.nvme1n1.in‚ 184.21875! disk.nvme0n1.out‚ 3059.2734375 disk.md0p1.in‚ 0.4296875 disk.sdb2.out‚ 89.2041015625 disk.md0.out‚ 86.37890625 proc.cpu.threads‚17 disk.dm-0.in‚ 477.53515625 disk.dm-0.out‚ 10393.78125 disk.sdb.out‚ 89.2041015625 disk.sdb1.out‚0 network.recv‚347979 disk.sda1.in‚0 disk.md0p1.out‚ 86.37890625 disk.sda.in‚ 0.27734375% memory_percent‚36.835646629333496 network.sent‚982379) disk./.usagePercent‚83.13443284413978% disk./.usageGB‚2973.2900009155273* proc.memory.availableMB‚10348.84765625 disk.sda2.in‚ 0.27734375 disk.sda.out‚ 89.2041015625 disk.sda1.out‚0 disk.sdb2.in‚ 0.15234375 disk.md0.in‚ 0.4296875 disk.rbd0.in‚0.015625 disk.sdb.in‚ 0.15234375# proc.memory.rssMB‚ 2932.14453125 cpu‚24.735728601765736 disk.rbd0.out‚ 0.279296875Çÿ#œÆ‚(:‘ ’çÔŠ­œ‡ gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚110.681' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚27.670250000000003 gpu.0.smClock‚1245 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EU´înÇ‚(:ã ’çÔøžˆ disk.sdb1.out‚0* proc.memory.percent‚17.896389961242676 disk.sda1.in‚0 disk.md0p1.out‚ 88.1171875 disk.rbd2.in‚0 disk.sdb.in‚ 0.15234375 disk.sdb.out‚90.99951171875* proc.memory.availableMB‚10348.83203125# proc.memory.rssMB‚ 2932.14453125 cpu‚24.740588095584624 network.sent‚ 1.034128e+06$ disk./.usageGB‚2973.665271759033 disk.md0p1.in‚ 0.4296875 disk.md0.in‚ 0.4296875 disk.sda.in‚ 0.27734375 network.recv‚353090 disk.sda2.in‚ 0.27734375 disk.sdb2.in‚ 0.15234375 disk.sdb2.out‚90.99951171875 disk.md0.out‚ 88.1171875 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 0.279296875 disk.sdb1.in‚0 disk.sda.out‚90.99951171875 disk.nvme0n1.in‚ 293.31640625 disk.nvme1n1.in‚ 184.21875" disk.nvme0n1.out‚ 3213.72265625 disk.dm-0.in‚ 477.53515625$ memory_percent‚36.83574199676514 proc.cpu.threads‚17 disk.sda2.out‚90.99951171875" disk.nvme1n1.out‚ 7334.58203125) disk./.usagePercent‚83.14492557398725 disk.sda1.out‚0 disk.rbd2.out‚ 0.279296875 disk.dm-0.out‚ 10548.3046875¬ôốÈ‚(:† ¡çÔ¢Óç‡ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚92.689' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚23.17225 gpu.0.smClock‚1215 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"‘ÛÐbÉ‚(:× ¡çÔ¼‹Ÿˆ network.recv‚357401 disk.sda1.in‚0 disk.md0p1.in‚ 0.4296875 disk.sdb.in‚ 0.15234375 disk.md0.out‚ 89.31640625 disk.rbd0.in‚0.015625# proc.memory.rssMB‚ 2932.14453125 disk.sdb1.out‚0 disk.sda.out‚ 92.255859375 disk.nvme0n1.out‚ 3471.8125 disk.sda2.in‚ 0.27734375 disk.md0.in‚ 0.4296875 disk.rbd2.out‚ 0.279296875! disk.nvme1n1.out‚ 7334.7109375 disk.sdb.out‚ 92.255859375 disk.sdb1.in‚0 disk.dm-0.in‚ 477.53515625 disk.sdb2.out‚ 92.255859375$ memory_percent‚36.83733940124512) disk./.usagePercent‚83.15891996910429 disk.sda2.out‚ 92.255859375 disk.md0p1.out‚ 89.31640625 disk.nvme1n1.in‚ 184.21875* proc.memory.percent‚17.896389961242676 proc.cpu.threads‚17 network.sent‚ 1.03783e+06% disk./.usageGB‚2974.1657791137695 disk.nvme0n1.in‚ 293.31640625 disk.dm-0.out‚ 10806.5234375 disk.sda1.out‚0 disk.sda.in‚ 0.27734375) proc.memory.availableMB‚ 10348.5703125 cpu‚24.745531313797095 disk.sdb2.in‚ 0.15234375 disk.rbd0.out‚ 0.279296875 disk.rbd2.in‚0€„År‘Ê‚(:† °çÔ®”Ú‡ gpu.0.gpu‚26.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚135.802' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.9505 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Å¿˜¨bË‚(:× °çÔ¹¼ˆ proc.cpu.threads‚17 network.sent‚ 1.04607e+06 disk.sda.in‚ 0.27734375 disk.sdb2.out‚ 93.935546875 disk.sdb1.in‚0 disk.sdb2.in‚ 0.15234375 disk.rbd2.in‚0! disk.nvme1n1.out‚ 7334.8828125 disk.dm-0.in‚ 477.53515625& proc.memory.availableMB‚ 10348.3125 disk.sda1.in‚0( disk./.usagePercent‚83.1588921307054$ disk./.usageGB‚2974.164783477783 disk.rbd2.out‚ 0.279296875 disk.nvme1n1.in‚ 184.21875 disk.sdb1.out‚0 disk.nvme0n1.in‚ 293.31640625" disk.nvme0n1.out‚ 3626.43359375 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 0.279296875 disk.sdb.in‚ 0.15234375 disk.dm-0.out‚10961.31640625 disk.sda2.out‚ 93.935546875 disk.md0p1.out‚ 90.94140625$ memory_percent‚36.83891296386719* proc.memory.percent‚17.896389961242676 disk.sda1.out‚0 disk.sda2.in‚ 0.27734375 disk.md0.out‚ 90.94140625# proc.memory.rssMB‚ 2932.14453125 cpu‚24.750019974529994 disk.md0.in‚ 0.4296875 disk.md0p1.in‚ 0.4296875 network.recv‚361664 disk.sdb.out‚ 93.935546875 disk.sda.out‚ 93.935546875›à•0Ì+ _wandbruntime‚874  _runtime‚874uDð8Ãͽ & lr-AdamW/pg1‚0.0005107651562495603 & lr-AdamW/pg2‚0.0005107651562495603 trainer/global_step‚749 $ _timestamp‚1.788464820839184e+09 _runtime‚ 874.024169412 _step‚56JæåÇÎÁ & lr-AdamW/pg1‚0.0005107651562495603 & lr-AdamW/pg2‚0.0005107651562495603 trainer/global_step‚749 $ _timestamp‚1.788464820839184e+09 _runtime‚ 874.024169412 _step‚5684ý.0Ï+ _wandbruntime‚874  _runtime‚874ºSµâÁ'лO YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚32779705096704 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1468228997944 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1472163602944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20082674 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5226861 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚700 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚30954676057600 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚30954425153024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1472163602944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13337081 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 315326464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚29629603047144 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13718739 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1472396504576 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚702 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚32780020423168 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13336379 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302147072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 291386368 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1472396504576 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31280076997120 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚33776567 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚33801415 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13718741 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚29482261550080 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 329989632 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚29482279553024 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 trainer/global_step‚749 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1499628099584 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚700 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚30954676057600 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚30954425153024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1472163602944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13337081 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 315326464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚29629603047144 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13718739 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1472396504576 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚702 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚32780020423168 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13336379 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302147072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 291386368 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1472396504576 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31280076997120 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚33776567 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚33801415 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13718741 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚29482261550080 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 329989632 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚29482279553024 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 trainer/global_st%á¹` ep‚749 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1499628099584 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚29521547587072 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5233435 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13737014 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20084520 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31321760967168 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚29669091893762 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33846408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13354457 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚698 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1501911303680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1470187487138 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1501619930624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1469958513394 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13353759 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5232739 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13737016 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 291373056 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20109392 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33821534 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚28198904406624 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚30995926133760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1474360543744 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 _step‚58 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33846408 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20084520 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚30995675215872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚32823696210944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 315313152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚749 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508f 4tÏ&ÔÉM D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1501619930624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1469958513394 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13353759 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5232739 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13737016 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 291373056 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20109392 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33821534 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚28198904406624 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚30995926133760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1474360543744 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 _step‚58 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33846408 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20084520 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚30995675215872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚32823696210944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 315313152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚749 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8121022 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302147072 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚30995926133760 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1474127628800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚29521547587072 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5233435 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13737014 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20084520 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31321760967168 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚29669091893762 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33846408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13354457 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚698 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1501911303680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1470187487138 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0:¥€0Õ+ _wandbruntime‚875  _runtime‚875Á½o¸PÖÊ .loss/FBt2/detection_loss‚6.643842816352844 trainer/global_step‚749 /loss/FRt2/assignment_loss‚5.711931896209717 /loss/FRt4/assignment_loss‚6.253861090716193 0loss/FBt1/assignment_loss‚0.6608558946422168 1loss/SRqqt1/detection_loss‚2.7424856747899735 1loss/SRqqt4/detection_loss‚2.7922598847321103 1loss/FBt2/assignment_loss‚0.47769534587831686 .loss/FRt1/detection_loss‚2.803308046696817 .loss/FRt2/detection_loss‚2.788485956192017 _runtime‚ 875.215123561 /loss/FRt1/assignment_loss‚5.993226640364703 .loss/FBt1/detection_loss‚4.508569802556719 epoch‚0 0loss/FBt4/assignment_loss‚0.2913446262245998 /loss/FRt4/detection_loss‚3.7291574898888085 %loss/total_loss‚70.98000265266352 % _timestamp‚1.7884648220303717e+09 1loss/SRqqt1/assignment_loss‚2.382514148099082 2loss/SRqqt3/assignment_loss‚2.5460739795005685 2loss/SRqqt4/assignment_loss‚3.0556980073451996 0loss/FBt3/assignment_loss‚0.3404654284947416 0loss/SRqqt2/detection_loss‚2.786380029283464 1loss/SRqqt3/detection_loss‚1.0263932803739804 .loss/FBt4/detection_loss‚4.156740348128712 2loss/SRqqt2/assignment_loss‚3.0194799341261387 .loss/FBt3/detection_loss‚0.837167271561096 _step‚59 /loss/FRt3/assignment_loss‚3.734169835463548 .loss/FRt3/detection_loss‚1.697895225041952DVeT×Î /loss/FRt2/assignment_loss‚5.711931896209717 /loss/FRt4/assignment_loss‚6.253861090716193 0loss/FBt1/assignment_loss‚0.6608558946422168 1loss/SRqqt1/detection_loss‚2.7424856747899735 1loss/SRqqt4/detection_loss‚2.7922598847321103 1loss/FBt2/assignment_loss‚0.47769534587831686 .loss/FRt1/detection_loss‚2.803308046696817 .loss/FRt2/detection_loss‚2.788485956192017 _runtime‚ 875.215123561 /loss/FRt1/assignment_loss‚5.993226640364703 .loss/FBt1/detection_loss‚4.508569802556719 epoch‚0 0loss/FBt4/assignment_loss‚0.2913446262245998 /loss/FRt4/detection_loss‚3.7291574898888085 %loss/total_loss‚70.98000265266352 % _timestamp‚1.7884648220303717e+09 1loss/SRqqt1/assignment_loss‚2.382514148099082 2loss/SRqqt3/assignment_loss‚2.5460739795005685 2loss/SRqqt4/assignment_loss‚3.0556980073451996 0loss/FBt3/assignment_loss‚0.3404654284947416 0loss/SRqqt2/detection_loss‚2.786380029283464 1loss/SRqqt3/detection_loss‚1.0263932803739804 .loss/FBt4/detection_loss‚4.156740348128712 2loss/SRqqt2/assignment_loss‚3.0194799341261387 .loss/FBt3/detection_loss‚0.837167271561096 _step‚59 /loss/FRt3/assignment_loss‚3.734169835463548 .loss/FRt3/detection_loss‚1.697895225041952 .loss/FBt2/detection_loss‚6.643842816352844 trainer/global_step‚749;˜¿òL‘Ø‚(:† ¿çÔó‰Ò‡ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚133.626' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.4065 gpu.0.smClock‚1245 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½6eZÙ‚(:Ï ¿çÔן׈ disk.md0p1.out‚ 92.609375 disk.sdb2.in‚ 0.15234375 disk.rbd2.out‚ 0.279296875 disk.nvme0n1.in‚ 293.31640625 disk.sda1.in‚0 disk.rbd0.in‚0.015625* proc.memory.percent‚17.896413803100586 proc.cpu.threads‚17% disk./.usageGB‚2974.6654090881348 disk.nvme1n1.in‚ 184.21875 disk.nvme1n1.out‚7335 disk.sda.in‚ 0.27734375 disk.rbd2.in‚0 disk.sdb.in‚ 0.15234375 disk.sdb1.out‚0 disk.dm-0.out‚ 11182.3515625 disk.sda1.out‚0 disk.sdb2.out‚ 95.6552734375) disk./.usagePercent‚83.17288983229892 disk.sdb1.in‚0 disk.sda.out‚ 95.6552734375" proc.memory.rssMB‚ 2932.1484375 network.recv‚365585! disk.nvme0n1.out‚ 3847.3515625 disk.dm-0.in‚ 477.53515625 disk.sda2.out‚ 95.6552734375 network.sent‚ 1.062037e+06 disk.md0p1.in‚ 0.4296875 disk.rbd0.out‚ 0.279296875$ memory_percent‚36.83910369873047 cpu‚24.75463552654723 disk.sdb.out‚ 95.6552734375 disk.sda2.in‚ 0.27734375 disk.md0.in‚ 0.4296875 disk.md0.out‚ 92.609375' proc.memory.availableMB‚ 10348.28125VËWì’Ú‚(:‡ ÎçÔÀ½‡ gpu.0.gpu‚42.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚135.517' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.87925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V\ªƒjÛ‚(:ß ÎçÔ«„ð‡ disk.sdb1.in‚0 disk.dm-0.out‚11432.86328125 disk.sda2.out‚ 97.1552734375" proc.memory.rssMB‚ 2932.1484375 cpu‚24.759158893898345$ disk./.usageGB‚2974.133140563965 disk.sda1.out‚0 disk.md0.out‚ 94.05078125* proc.memory.percent‚17.896413803100586 network.recv‚370834 disk.sdb1.out‚0" disk.nvme0n1.out‚ 4097.72265625 disk.md0.in‚ 0.4296875$ memory_percent‚36.83903217315674 disk.sdb.out‚ 97.1552734375 disk.rbd0.in‚0.015625 disk.dm-0.in‚ 477.53515625 disk.sdb2.out‚ 97.1552734375 disk.nvme0n1.in‚ 293.31640625 disk.md0p1.out‚ 94.05078125 disk.rbd0.out‚ 0.279296875 disk.sda.out‚ 97.1552734375 disk.sda1.in‚0 disk.sda2.in‚ 0.27734375 disk.sdb2.in‚ 0.15234375* proc.memory.availableMB‚10348.29296875 proc.cpu.threads‚17 network.sent‚ 1.100756e+06) disk./.usagePercent‚83.15800738159145 disk.nvme1n1.in‚ 184.21875 disk.nvme1n1.out‚ 7335.140625 disk.sda.in‚ 0.27734375 disk.md0p1.in‚ 0.4296875 disk.rbd2.in‚0 disk.rbd2.out‚ 0.279296875 disk.sdb.in‚ 0.15234375G€Pª’Ü‚(:‡ ÝçÔÉñ”‡ gpu.0.gpu‚42.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚139.251' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.81275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<ÁÉ[Ý‚(:Ð ÝçÔËêä‡ disk.sdb2.in‚ 0.15234375 disk.rbd2.in‚0 disk.sda.in‚ 0.27734375 disk.dm-0.in‚ 477.53515625 disk.dm-0.out‚11615.33984375 disk.sda2.in‚ 0.27734375* proc.memory.percent‚17.896413803100586 cpu‚24.763500256088406 disk.sdb2.out‚ 98.3515625 disk.sda2.out‚ 98.3515625 disk.md0p1.out‚ 95.19921875" disk.nvme0n1.out‚ 4280.07421875 disk.sda1.out‚0 disk.md0.out‚ 95.19921875 disk.md0.in‚ 0.4296875 disk.md0p1.in‚ 0.4296875 disk.nvme1n1.out‚ 7335.265625% disk./.usageGB‚2974.1328735351562 disk.sdb.in‚ 0.15234375 disk.sda1.in‚0" proc.memory.rssMB‚ 2932.1484375 proc.cpu.threads‚17 network.sent‚ 1.106699e+06 disk.sdb1.out‚0 disk.sda.out‚ 98.3515625 disk.rbd2.out‚ 0.279296875 disk.nvme1n1.in‚ 184.21875 network.recv‚375345( disk./.usagePercent‚83.1579999153542 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 0.279296875 disk.nvme0n1.in‚ 293.31640625$ memory_percent‚36.83910369873047 disk.sdb.out‚ 98.3515625 disk.sdb1.in‚0' proc.memory.availableMB‚ 10348.28125ÊY|Þ‚(:… ìçÔÞÁ‡ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚142.93' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.7325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jhß‚(:Ý ìçÔøµñ‡ disk.sda.in‚ 0.27734375 disk.sdb2.out‚100.4033203125 disk.md0.in‚ 0.4296875 disk.md0.out‚ 97.1953125 disk.rbd0.out‚ 0.279296875 disk.rbd2.out‚ 0.279296875 disk.sda1.in‚0 proc.cpu.threads‚17 network.sent‚ 1.112683e+06 network.recv‚379641 disk.md0p1.in‚ 0.4296875 disk.md0p1.out‚ 97.1953125" proc.memory.rssMB‚ 2932.1484375* proc.memory.percent‚17.896413803100586 cpu‚24.7676966502699 disk.nvme1n1.in‚ 184.21875" disk.nvme1n1.out‚ 7335.51953125 disk.sdb1.out‚0# memory_percent‚36.8391752243042* proc.memory.availableMB‚10348.26953125 disk.sdb.out‚100.4033203125 disk.sdb1.in‚0" disk.nvme0n1.out‚ 4413.82421875 disk.sdb.in‚ 0.15234375 disk.nvme0n1.in‚ 293.31640625 disk.dm-0.in‚ 477.53515625 disk.dm-0.out‚ 11749.34375 disk.sda2.in‚ 0.27734375 disk.sda2.out‚100.4033203125) disk./.usagePercent‚83.15658399679029 disk.sda1.out‚0 disk.sda.out‚100.4033203125$ disk./.usageGB‚2974.082233428955 disk.sdb2.in‚ 0.15234375 disk.rbd0.in‚0.015625 disk.rbd2.in‚0p®0à+ _wandbruntime‚931  _runtime‚931î51Äá¾ & lr-AdamW/pg1‚0.0005448616286293709 & lr-AdamW/pg2‚0.0005448616286293709 trainer/global_step‚799 % _timestamp‚1.7884648784926617e+09 _runtime‚ 931.677615772 _step‚60ä` ÕÈâ _runtime‚ 931.677615772 _step‚60 & lr-AdamW/pg1‚0.0005448616286293709 & lr-AdamW/pg2‚0.0005448616286293709 trainer/global_step‚799 % _timestamp‚1.7884648784926617e+09< m›0ã+ _wandbruntime‚931  _runtime‚931òÙ-¾ä»O H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 _step‚61 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚30037874881040 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8650320 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1570612211200 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1570612211200 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 M@Devicš›ï$eStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14223829 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 309035008 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚33016942709248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14632489 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21420626 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5573511 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚31604058958852 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚31603812945956 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1565938064916 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚700 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21420626 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚31446563399680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚702 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚33016942709248 _runtime‚ 931.679006008 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8650318 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14224531 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5574211 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1570379309568 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14632491 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 329989632 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚30037891920400 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚31446581402624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1566167038452 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884648784937618e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14632489 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚34964190096896 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚31446563399680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚33364275499520 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36028269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚799 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚34963881061888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 285094912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302147072 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1570379309568 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21395780 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1570379309568 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21395780 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 _step‚61 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚30037874881040 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8650320 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1570612211200 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1570612211200 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14223829 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 309035008 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚33016942709248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14632489 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21420626 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5573511 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚31604058958852 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚31603812945956 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1565938064916 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚700 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21420626 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚31446563399680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚702 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚33016942709248 _runtime‚ 931.679006008 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8650318 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14224531 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5574211 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1570379309568 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14632491 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 329989632 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚30037891920400 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0=Í*͘0æ+ _wandbruntime‚933  _runtime‚933ÂW{Ë&çÅM \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1568125527646 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚33058192772096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8661122 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚31643547805470 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36073236 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1572576250368 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21447344 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36098110 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚ 933.219482718 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚31485867439616 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14650766 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5581355 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚30075405238464 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21422472 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8661120 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21447344 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚31485849436672 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14650766 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14650764 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302147072 _step‚62 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14241795 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21422472 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1601909977088 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280887296 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1601629089792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36098110 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚682 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚33405983409664 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚33405959469568 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚35007893386752 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚31643301792366 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 trainer/global_step‚799 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 % _timestamp‚1.7884648800337734e+09 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36073236 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14241795 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21422472 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1601909977088 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280887296 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1601629089792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:De_npµviceStatsMonitor.on_train_batch_end/active.all.allocated‚36098110 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚682 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚33405983409664 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚33405959469568 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚35007893386752 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚31643301792366 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 trainer/global_step‚799 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 % _timestamp‚1.7884648800337734e+09 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36073236 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8661122 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚31643547805470 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36073236 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1572576250368 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21447344 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36098110 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚ 933.219482718 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚31485867439616 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14650766 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5581355 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚30075405238464 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21422472 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8661120 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21447344 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚31485849436672 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14650766 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14650764 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302147072 _step‚62 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816>Kù 0é+ _wandbruntime‚933  _runtime‚933Q úDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚33410883252224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302147072 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚33410883252224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1668595016192 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15546239 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚37148048699904 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚35448497942016 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22733732 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚35079460265472 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1668827917824 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15546241 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306937856 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚35448474001920 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1699857695744 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚33410865249280 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚38304819 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15111073 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9190418 % _timestamp‚1.7884649367129676e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚33410865249280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚704 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚35079460265472 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15111777 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282997760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15546241 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚33578514870560 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5920655 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608ò»âU÷¿O ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚37148355637760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22733732 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚35079460265472 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1668827917824 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15546241 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306937856 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚35448474001920 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1699857695744 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚33410865249280 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚38304819 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15111073 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9190418 % _timestamp‚1.7884649367129676e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚33410865249280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚704 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚35079460265472 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15111777 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282997760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15546241 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚33578514870560 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5920655 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚31914409791600 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚35079711170048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 329989632 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1663876105424 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1699574697984 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 _step‚65 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1668595016192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1668827917824 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚31914392752240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22758578 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1664105078960 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9190420 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_trkÕ#pain_batch_start/active.small_pool.peak‚30508 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚33410883252224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302147072 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚33410883252224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1668595016192 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15546239 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚37148048699904 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚35448497942016 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15128672 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1665834594410 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚35490157971968 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚35120710328320 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1701577295360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15564514 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5928133 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚66 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚33450169289216 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _runtime‚ 991.081213533 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1670791956992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚35490181912064 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚33450151286272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5927452 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚37192035900416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1670559042048 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚35120961246208 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚33450169289216 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 329989632 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1670559042048 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38349812 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38324938 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608¾¥#æÏ&üÉM VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚35120961246208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _runtime‚ 991.081213533 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1670791956992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚35490181912064 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚33450151286272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5927452 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚37192035900416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1670559042048 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚35120961246208 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚33450169289216 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 329989632 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1670559042048 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38349812 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38324938 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚31951940149024 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15128672 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1665834594410 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚35490157971968 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚35120710328320 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1701577295360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15564514 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5928133 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚66 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚33450169289216 Fjƒ‚(:ß ÆžçÔ튈 disk.sdb1.in‚0" disk.nvme0n1.out‚ 5639.69921875 disk.sda2.in‚ 0.27734375 disk.sda2.out‚ 110.158203125$ memory_percent‚36.84046268463135 disk.rbd0.out‚ 0.279296875 disk.rbd0.in‚0.015625 disk.rbd2.in‚0* proc.memory.availableMB‚10348.05859375 disk.dm-0.out‚12976.54296875( disk./.usagePercent‚83.1728336221985 disk.nvme0n1.in‚ 293.31640625 disk.dm-0.in‚ 477.53515625 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.896437644958496 network.sent‚ 1.254531e+06 network.recv‚406111$ disk./.usageGB‚2974.663398742676 disk.sdb2.out‚ 110.158203125 disk.md0.out‚ 106.59765625 disk.sdb.out‚ 110.158203125 disk.sda.in‚ 0.27734375 disk.sda.out‚ 110.158203125 disk.md0.in‚ 0.4296875 disk.nvme1n1.out‚ 7336.84375 disk.sdb1.out‚0 disk.md0p1.in‚ 0.4296875 cpu‚24.78990734717527 proc.cpu.threads‚17 disk.nvme1n1.in‚ 184.21875 disk.sdb2.in‚ 0.15234375 disk.rbd2.out‚ 0.279296875 disk.sdb.in‚ 0.15234375 disk.md0p1.out‚ 106.59765625# proc.memory.rssMB‚ 2932.15234375ÌP> ’„‚(:‡ ÕžçÔÇ gpu.0.gpu‚47.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚132.399' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.09975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹¼þk…‚(:à ÕžçÔ·¹ªˆ! disk.nvme1n1.out‚ 7336.9765625 disk.dm-0.out‚13256.18359375 disk.sda1.out‚0 cpu‚24.793080579699133 proc.cpu.threads‚17 disk.md0.in‚ 0.4296875 disk.sdb.in‚ 0.15234375 disk.sdb.out‚ 111.439453125 disk.sda1.in‚0 disk.sdb2.in‚ 0.15234375$ memory_percent‚36.84048652648926) proc.memory.availableMB‚ 10348.0546875 disk.rbd2.in‚0 disk.sda2.out‚ 111.439453125 disk.nvme1n1.in‚ 184.21875 disk.nvme0n1.in‚ 293.31640625 disk.sdb2.out‚ 111.439453125 disk.md0p1.in‚ 0.4296875 network.sent‚ 1.26033e+06 disk.md0.out‚ 107.82421875 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 0.279296875" disk.nvme0n1.out‚ 5919.20703125( disk./.usagePercent‚83.1728336221985$ disk./.usageGB‚2974.663398742676 disk.sda.in‚ 0.27734375 disk.sda2.in‚ 0.27734375# proc.memory.rssMB‚ 2932.15234375* proc.memory.percent‚17.896437644958496 disk.rbd0.out‚ 0.279296875 disk.dm-0.in‚ 477.53515625 disk.md0p1.out‚ 107.82421875 network.recv‚410398 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda.out‚ 111.439453125^ÀÿÚ2†- _wandbruntime‚1048  _runtime‚1048¨lfˆÅ‡¿ & lr-AdamW/pg1‚0.0006130545733889917 & lr-AdamW/pg2‚0.0006130545733889917 trainer/global_step‚899 % _timestamp‚1.7884649950025363e+09 _runtime‚1048.187509151 _step‚68ÅóS"Ɉà _runtime‚1048.187509151 _step‚68 & lr-AdamW/pg1‚0.0006130545733889917 & lr-AdamW/pg2‚0.0006130545733889917 trainer/global_step‚899 % _timestamp‚1.7884649950025363e+09D»lra2‰- _wandbruntime‚1048  _runtime‚1048ô¬¤xkŠ¼O OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1766810722816 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15994810 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚33790910623440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1762043c6«ÐW#119468 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚682 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302147072 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 % _timestamp‚1.7884649950036814e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 _step‚69 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24071684 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40556521 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚37532696444416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1767043624448 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1761814145932 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 trainer/global_step‚899 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282997760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15995494 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1766810722816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16459991 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24071684 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24096530 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9730520 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚39332499005952 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37141977821696 _runtime‚1048.188738916 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6264974 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1799519563776 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37142228726272 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16459991 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16459989 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1799802561536 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306937856 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚37532672504320 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6264292 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582©`(HÆ'‹ÀO M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40556521 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚37532696444416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1767043624448 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1761814145932 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 trainer/global_step‚899 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282997760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15995494 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1766810722816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16459991 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24071684 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24096530 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9730520 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚39332499005952 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37141977821696 _runtime‚1048.188738916 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6264974 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1799519563776 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37142228726272 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16459991 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16459989 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1799802561536 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306937856 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚37532672504320 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6264292 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1766810722816 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15994810 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚33790910623440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1762043119468 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚682 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302147072 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 % _timestamp‚1.7884649950036814e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 _step‚69 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24071684 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0E÷þ¹¬2Œ- _wandbruntime‚1049  _runtime‚1049Øf=àÌ&ÆM UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6271774 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24123248 trainer/global_step‚899 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚35592213615782 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 % _timestamp‚1.7884649959714663e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9741322 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302147072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚35414453135872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1801807592448 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24123248 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1769007663616 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24098376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6271101 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40601514 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16012421 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1769007663616 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚35414471138816 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚33828440980864 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚35414453135872 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 _runtime‚1049.157208211 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16013096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1801522510848 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1768774748672 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37183478802432 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9741320 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16478264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚35414471138816 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚35592459628886 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚33828458020224 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39376188006912 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16478264 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24098376 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚39375878985216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1764001608662 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40601514 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚37574380414464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40601514 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚37574380414464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24123248 trainer/global_step‚899 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚35592213615782 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 % _timestamp‚1.7884649959714663e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9741322 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302147072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚35414453135872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1801807592448 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24123248 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1769007663616 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24098376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6271101 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40601514 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16012421 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1769007663616 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚35414471138816 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚33828440980864 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚35414453135872 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 _runtime‚1049.157208211 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16013096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1801522510848 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1768774748672 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37183478802432 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9741320Fƒ‰ æ2- _wandbruntime‚1049  _runtime‚1049 òœ1QË trainer/global_step‚899 .loss/FRt2/assignment_loss‚5.50583207285082 1loss/SRqqt2/detection_loss‚2.6248652509280612 1loss/SRqqt3/detection_loss‚0.7643497910188592 /loss/FBt4/detection_loss‚2.6410037734929253 1loss/SRqqt3/assignment_loss‚2.023830913655136 /loss/FRt1/detection_loss‚2.6704761527478693 .loss/FRt4/detection_loss‚3.539815303825197 0loss/SRqqt1/detection_loss‚2.175793086157905 epoch‚0 %loss/total_loss‚63.51613167875737 1loss/SRqqt2/assignment_loss‚2.374920345204217 0loss/FBt2/assignment_loss‚0.5774801816636076 /loss/FRt3/detection_loss‚1.8504446788745768 _step‚71 /loss/FRt1/assignment_loss‚5.298216879367828 0loss/FBt4/assignment_loss‚0.8584667759783128 1loss/SRqqt4/detection_loss‚3.0021970100127735 2loss/SRqqt1/assignment_loss‚2.3182282868358826 /loss/FBt3/detection_loss‚0.8172598263236783 $ _timestamp‚1.788464995974336e+09 _runtime‚1049.159060289 .loss/FBt1/detection_loss‚4.056875248750051 .loss/FBt2/detection_loss‚4.712777614593506 /loss/FRt2/detection_loss‚2.5763970516823433 0loss/FRt3/assignment_loss‚3.2774950861930847 /loss/FRt4/assignment_loss‚6.508749008178711 0loss/FBt1/assignment_loss‚0.7190245650708297 1loss/FBt3/assignment_loss‚0.35471749297336286 0loss/SRqqt4/assignment_loss‚2.26691528237783^­þU‘Ï .loss/FBt1/detection_loss‚4.056875248750051 .loss/FBt2/detection_loss‚4.712777614593506 /loss/FRt2/detection_loss‚2.5763970516823433 0loss/FRt3/assignment_loss‚3.2774950861930847 /loss/FRt4/assignment_loss‚6.508749008178711 0loss/FBt1/assignment_loss‚0.7190245650708297 1loss/FBt3/assignment_loss‚0.35471749297336286 0loss/SRqqt4/assignment_loss‚2.26691528237783 trainer/global_step‚899 .loss/FRt2/assignment_loss‚5.50583207285082 1loss/SRqqt2/detection_loss‚2.6248652509280612 1loss/SRqqt3/detection_loss‚0.7643497910188592 /loss/FBt4/detection_loss‚2.6410037734929253 1loss/SRqqt3/assignment_loss‚2.023830913655136 /loss/FRt1/detection_loss‚2.6704761527478693 .loss/FRt4/detection_loss‚3.539815303825197 0loss/SRqqt1/detection_loss‚2.175793086157905 epoch‚0 %loss/total_loss‚63.51613167875737 1loss/SRqqt2/assignment_loss‚2.374920345204217 0loss/FBt2/assignment_loss‚0.5774801816636076 /loss/FRt3/detection_loss‚1.8504446788745768 _step‚71 /loss/FRt1/assignment_loss‚5.298216879367828 0loss/FBt4/assignment_loss‚0.8584667759783128 1loss/SRqqt4/detection_loss‚3.0021970100127735 2loss/SRqqt1/assignment_loss‚2.3182282868358826 /loss/FBt3/detection_loss‚0.8172598263236783 $ _timestamp‚1.788464995974336e+09 _runtime‚1049.159060289G#vå ’’‚(:‡ äžçÔ¸½‡ gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚139.251' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.81275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04À%m“‚(:â äžçԱϊˆ network.recv‚414556 disk.rbd0.in‚0.015625 disk.nvme1n1.in‚ 184.21875 disk.sdb.out‚113.3115234375# proc.memory.rssMB‚ 2932.15234375 disk.rbd2.in‚0! disk.nvme1n1.out‚ 7337.1171875 disk.sdb1.in‚0 disk.nvme0n1.in‚ 293.31640625 disk.sda2.in‚ 0.27734375 cpu‚24.796418527591616( disk./.usagePercent‚83.1728336221985 disk.rbd2.out‚ 0.279296875 disk.sda.in‚ 0.27734375" disk.nvme0n1.out‚ 6110.91796875* proc.memory.percent‚17.896437644958496 disk.sdb2.in‚ 0.15234375 disk.sdb2.out‚113.3115234375 disk.md0.in‚ 0.4296875 disk.dm-0.in‚ 477.53515625 disk.sda2.out‚113.3115234375 disk.md0p1.in‚ 0.4296875 network.sent‚ 1.276906e+06 proc.cpu.threads‚17$ disk./.usageGB‚2974.663398742676 disk.sdb1.out‚0 disk.sda1.in‚0 disk.md0p1.out‚ 109.64453125 disk.md0.out‚ 109.64453125 disk.rbd0.out‚ 0.279296875 disk.sdb.in‚ 0.15234375 disk.dm-0.out‚13448.03515625$ memory_percent‚36.84196472167969& proc.memory.availableMB‚ 10347.8125 disk.sda.out‚113.3115234375 disk.sda1.out‚0¾v`d’”‚(:‡ óžçÔ»¿‡ gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚139.251' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.81275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g×m•‚(:â óžçÔÍëó‡" disk.nvme1n1.out‚ 7337.26953125 disk.sdb.out‚114.8701171875 disk.sdb2.out‚114.8701171875 disk.rbd2.in‚0$ memory_percent‚36.84060573577881 disk.nvme0n1.in‚ 293.31640625 disk.sda2.out‚114.8701171875 proc.cpu.threads‚17 network.sent‚ 1.314555e+06) disk./.usagePercent‚83.15800236854643% disk./.usageGB‚2974.1329612731934 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0p1.in‚ 0.4296875 disk.md0.in‚ 0.4296875 disk.sda.in‚ 0.27734375 disk.dm-0.in‚ 477.53515625 disk.rbd2.out‚ 0.279296875* proc.memory.availableMB‚10348.03515625 cpu‚24.799662003132514 disk.sdb1.in‚0 disk.dm-0.out‚13497.62109375 disk.sdb.in‚ 0.15234375 disk.sda.out‚114.8701171875! disk.nvme0n1.out‚ 6160.3515625 disk.md0.out‚ 111.15625 disk.rbd0.out‚ 0.279296875 network.recv‚418510 disk.nvme1n1.in‚ 184.21875 disk.sda2.in‚ 0.27734375 disk.rbd0.in‚0.015625* proc.memory.percent‚17.896437644958496 disk.md0p1.out‚ 111.15625 disk.sdb2.in‚ 0.15234375# proc.memory.rssMB‚ 2932.15234375¦ºŠk’–‚(:‡ ‚ŸçÔÛö‡ gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚124.661' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.16525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ò½ •n—‚(:ã ‚ŸçԷëˆ$ disk./.usageGB‚2974.082344055176 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 0.279296875 disk.sda2.in‚ 0.27734375 cpu‚24.802789748533243) disk./.usagePercent‚83.15658708994572 disk.sda.in‚ 0.27734375 disk.dm-0.out‚ 13697.8984375 disk.sda1.out‚0$ memory_percent‚36.84127330780029 disk.nvme1n1.in‚ 184.21875 network.recv‚423226 disk.nvme0n1.in‚ 293.31640625 disk.sdb2.in‚ 0.15234375! disk.sdb2.out‚116.32470703125 disk.md0.out‚ 112.546875* proc.memory.availableMB‚10347.92578125 disk.md0p1.in‚ 0.4296875 network.sent‚ 1.318696e+06 disk.rbd2.in‚0 disk.nvme1n1.out‚ 7337.40625 disk.sdb.out‚116.32470703125 disk.sdb1.out‚0! disk.nvme0n1.out‚ 6360.4921875 disk.dm-0.in‚ 477.53515625 disk.rbd0.out‚ 0.279296875 disk.sdb.in‚ 0.15234375 disk.sdb1.in‚0! disk.sda2.out‚116.32470703125 disk.md0p1.out‚ 112.546875 disk.md0.in‚ 0.4296875# proc.memory.rssMB‚ 2932.15234375* proc.memory.percent‚17.896437644958496 disk.sda.out‚116.32470703125 disk.sda1.in‚0 proc.cpu.threads‚17–DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1865259331072 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16882241 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚697 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17373741 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336281088 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚39204746282496 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25434482 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚39204746282496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚37339468948480 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42808223 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25409636 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚39616894946816 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1859981159976 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17373739 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚949 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚39204495377920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1865026429440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚39616871006720 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 _runtime‚1106.366779421 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚37339486951424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25434482 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10270620 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1899564626944 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚37527180681080 _step‚73 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10270618 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚35667445534000 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884650531813066e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1865026429440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 295580672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚37339468948480 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚37339486951424 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0\=\òŸÂO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1899564626944 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚37527180681080 _step‚73 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10270618 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚35667445534000 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884650531813066e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1865026429440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 295580672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚37339468948480 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚37339486951424 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16882940 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1899860207616 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚35667428494640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 319520768 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚41516435633664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17373739 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1865259331072 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16882241 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚697 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17373741 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 SEDeviceStatsMonitor.on_train_batch_start/£?v=Ö inactive_split_bytes.all.peak‚ 336281088 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚39204746282496 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25434482 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚39204746282496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚37339468948480 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42808223 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25409636 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚39616894946816 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1859981159976 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17373739 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚949 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚39204495377920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1865026429440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚39616871006720 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 _runtime‚1106.366779421 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚37339486951424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25434482 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10270620Iõ2œ)2 - _wandbruntime‚1107  _runtime‚1107¨šôÍ&¡ÇM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10281422 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6618907 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚39658578916864 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1901568016384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚37566915540594 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1901842612224 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚37378772988416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚688 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚37378772988416 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25461200 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 274595840 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25436328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16900329 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17392014 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42853216 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42828342 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42853216 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42828342 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17392014 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚39658554976768 trainer/global_step‚949 _step‚74 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚39245996358656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1866990455296 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚41560122993152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788465054137344e+09 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚35704958852064 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1867223370240 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6618221 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚37378754985472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1861939649170 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚41560421529088 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 298535936 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16899641 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚39245996358656 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1861710675426 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17392016 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚35704975891424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10281420 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25436328 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1107.323059897 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚37566669527490 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273¤Âä Ñ&¢ËM >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚35704958852064 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1867223370240 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6618221 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚37378754985472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1861939649170 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚41560421529088 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 298535936 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16899641 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚39245996358656 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1861710675426 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17392016 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚35704975891424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10281420 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25436328 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1107.323059897 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚37566669527490 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10281422 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6618907 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚39658578916864 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1901568016384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚37566915540594 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1901842612224 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚37378772988416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚688 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚37378772988416 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25461200 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 274595840 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25436328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16900329 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17392014 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42853216 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42828342 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42853216 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42828342 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17392014 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚39658554976768 trainer/global_step‚949 _step‚74 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚39245996358656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1866990455296 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚41560122993152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788465054137344e+09 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41267012934144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18287491 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚701 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26772434 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1963475037696 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1963242136064 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚43701068355584 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚39303788801024 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26747588 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1957690226948 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336281088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚37543946365840 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1957919200484 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26747588 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17769654 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10810720 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _runtime‚1164.746213742 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45059925 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚39303770798080 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6958936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚41267263838720 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚41267263838720 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 291386368•Ø¬òÇ'³ÁO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1963475037696 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1963242136064 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚43701068355584 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚39303788801024 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26747588 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1957690226948 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336281088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚37543946365840 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1957919200484 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26747588 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17769654 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10810720 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _runtime‚1164.746213742 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45059925 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚39303770798080 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6958936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚41267263838720 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚41267263838720 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 291386368 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41701093449216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41267012934144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18287491 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚701 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26772434 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1959877689678 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18305764 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17787685 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 287178752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚37581476723264 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26774280 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884651125846083e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1959648715934 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45080044 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45080044 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18305766 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43744430820352 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 trainer/global_step‚999 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10821520 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2001964519936 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45104918 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26799152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10821522 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17788361 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1965206161920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 311118848 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26799152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18305764 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚39343056835072 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚41742777419264 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1965206161920 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6966165 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43744741939200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚39541125439198 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚39541371452302 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚39343056835072 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚41308513914880 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1965439076864 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336281088 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2001677341184 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582]Þ¡­K¶ÌM ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 REDeviceStatsGr'‡Monitor.on_train_batch_end/allocation.small_pool.allocated‚26799152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10821522 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17788361 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1965206161920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 311118848 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26799152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18305764 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚39343056835072 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚41742777419264 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1965206161920 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6966165 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43744741939200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚39541125439198 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚39541371452302 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚39343056835072 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚41308513914880 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1965439076864 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336281088 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2001677341184 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6966839 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1959877689678 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18305764 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17787685 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 287178752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚37581476723264 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26774280 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884651125846083e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1959648715934 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45080044 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45080044 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18305766 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43744430820352 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 trainer/global_step‚999 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10821520 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2001964519936 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45104918 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B©`¾‚(:Õ úŸçÔ‹´³‡ disk.rbd2.out‚ 0.279296875 disk.nvme1n1.in‚ 184.21875 disk.dm-0.in‚ 477.53515625 disk.sda2.in‚ 0.27734375 disk.md0p1.out‚ 125.03125) disk./.usagePercent‚83.14328417505843 disk.sda.out‚ 129.23046875 disk.sda1.out‚0 network.recv‚458184* proc.memory.availableMB‚10347.42578125 proc.cpu.threads‚17 network.sent‚ 1.454184e+06! disk.nvme1n1.out‚ 8005.5078125 disk.nvme0n1.in‚ 293.31640625 disk.dm-0.out‚ 15476.2890625$ memory_percent‚36.84432506561279 disk.sdb1.out‚0 disk.md0p1.in‚ 0.4296875# proc.memory.rssMB‚ 2932.15234375 disk.rbd0.out‚ 0.279296875 disk.sdb.in‚ 0.15234375 disk.sdb.out‚ 129.23046875 disk.sdb1.in‚0 disk.sda2.out‚ 129.23046875* proc.memory.percent‚17.896437644958496 cpu‚24.82485454813197 disk.md0.in‚ 0.4296875 disk.md0.out‚ 125.03125 disk.rbd2.in‚0 disk.sda1.in‚0% disk./.usageGB‚2973.6065673828125 disk.sdb2.in‚ 0.15234375 disk.sdb2.out‚ 129.23046875 disk.rbd0.in‚0.015625 disk.sda.in‚ 0.27734375 disk.nvme0n1.out‚ 7470.78125Nm@’¿‚(:‡ úŸçÔÞÎ㇠gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚141.087' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.27175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›üó$À‚(:… ‰ çÔ¼ˆš† gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚139.528' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÞæØóuÁ‚(:ê ‰ çÔòÕ—‡* proc.memory.percent‚17.896437644958496 cpu‚24.827368627068438 disk.sda2.in‚ 0.27734375 disk.rbd0.in‚0.015625 proc.cpu.threads‚17 disk.md0.in‚ 0.4296875) disk./.usagePercent‚83.14326060308083 disk.dm-0.in‚ 477.57421875! disk.sdb2.out‚130.62841796875# proc.memory.rssMB‚ 2932.15234375 network.recv‚462195 disk.nvme1n1.in‚ 184.234375 disk.sdb2.in‚ 0.15234375 disk.sdb.out‚130.62841796875 disk.dm-0.out‚ 15620.34375 disk.sda1.in‚0 disk.sda.out‚130.62841796875" disk.nvme1n1.out‚ 8149.50390625 disk.sdb.in‚ 0.15234375" disk.nvme0n1.out‚ 7470.83984375 disk.sda1.out‚0! disk.sda2.out‚130.62841796875 disk.md0p1.in‚ 0.4296875 disk.md0p1.out‚ 126.36328125) proc.memory.availableMB‚ 10347.3984375 disk.md0.out‚ 126.36328125$ disk./.usageGB‚2973.605724334717 disk.rbd2.in‚0 disk.rbd2.out‚ 0.279296875 disk.sda.in‚ 0.27734375 disk.nvme0n1.in‚ 293.33984375 network.sent‚ 1.459983e+06 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd0.out‚ 0.279296875% memory_percent‚36.844491958618164 ¦2Â- _wandbruntime‚1216  _runtime‚1216µËæßÄþ % _timestamp‚1.7884651630239892e+09 _runtime‚1216.208975315 _step‚80 % lr-AdamW/pg1‚0.000715343990528423 % lr-AdamW/pg2‚0.000715343990528423 trainer/global_step‚1049½Ôì-ÈÄ % _timestamp‚1.7884651630239892e+09 _runtime‚1216.208975315 _step‚80 % lr-AdamW/pg1‚0.000715343990528423 % lr-AdamW/pg2‚0.000715343990528423 trainer/global_step‚1049PóÄʼ2Å- _wandbruntime‚1216  _runtime‚1216ÉÀ?Å'Æ¿O QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 289289216 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2099577829376 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19201239 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28110386 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚41476092504496 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2061457842688 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚43329530490368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚713 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28085540 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7307717 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28110386 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19201241 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19201239 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7308428 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚41476338517392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚43329530490368 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚41268072647680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚45885159070208 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2099867118592 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18658535 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚711 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚43785291951616 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚43785268011520 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚41268090650624 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚45884845840896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28110386 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19201241 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19201239 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7308428 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚41476338517392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚43329530490368 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚41268072647680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚45885159070208 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2099867118592 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18658535 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚711 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚43785291951616 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚43785268011520 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4126809d¶×Å]0650624 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚45884845840896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0Q¡;æÐ2È- _wandbruntime‚1217  _runtime‚1217•©.[Ï&ÉÉM ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚41307376687616 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2063654783488 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11361620 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚41515827364010 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2057815730186 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47331746 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18675920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚82 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚43371031471104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19219516 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚676 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43826975921664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19219514 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884651639816914e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47356620 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚41307376687616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7314300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45928525614592 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285081600 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚41307358684672 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2057586756442 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚45928834636288 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2063654783488 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19219516 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11361622 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2101573633024 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 trainer/global_step‚1049 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28137104 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚41307358684672 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚43826951981568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚39458011633824 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2063421868544 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47356620 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚41515581350906 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47331746 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚07G‘aÓ&ÊÍM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884651639816914e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47356620 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚41307376687616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7314300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45928525614592 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285081600 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚41307358684672 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2057586756442 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚45928834636288 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2063654783488 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19219516 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11361622 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2101573633024 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 trainer/global_step‚1049 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28137104 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚41307358684672 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚43826951981568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚39458011633824 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2063421868544 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47356620 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚41515581350906 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47331746 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚41307376687616 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2063654783488 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11361620 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚41515827364010 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2057815730186 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47331746 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18675920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚82 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚43371031471104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19219516 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚676 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43826975921664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19219514 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872RÕLš2Ë- _wandbruntime‚1217  _runtime‚1217å4þ³SÌÍ 0loss/FBt2/assignment_loss‚0.2140019008948002 _step‚83 2loss/SRqqt2/assignment_loss‚2.0325890015000883 2loss/SRqqt3/assignment_loss‚2.0481112748384476 0loss/FBt3/assignment_loss‚0.3105147234897034 /loss/FRt2/detection_loss‚2.4990482184945084 0loss/SRqqt2/detection_loss‚3.009302564289259 .loss/FBt3/detection_loss‚0.823603900270926 /loss/FRt2/assignment_loss‚5.671828525822337 /loss/FRt3/assignment_loss‚2.969424213233747 2loss/SRqqt4/assignment_loss‚2.4061553295581573 .loss/FRt4/detection_loss‚3.553209053842645 0loss/FBt1/assignment_loss‚0.5820515835657716 1loss/SRqqt3/detection_loss‚1.1478429412175166 % _timestamp‚1.7884651639854496e+09 /loss/FRt4/assignment_loss‚6.134607365256862 0loss/FBt4/assignment_loss‚0.5408336454597702 1loss/SRqqt1/detection_loss‚1.9060835144840753 /loss/FBt1/detection_loss‚3.6707089096307755 /loss/FBt2/detection_loss‚4.5162153244018555 .loss/FBt4/detection_loss‚2.685789752751589 trainer/global_step‚1049 /loss/FRt1/assignment_loss‚4.102292206552294 -loss/FRt1/detection_loss‚2.35581766433186 epoch‚0 2loss/SRqqt1/assignment_loss‚2.1696199098458657 /loss/FRt3/detection_loss‚1.6096987128257751 0loss/SRqqt4/detection_loss‚2.766214886019307 %loss/total_loss‚59.72556512257793 _runtime‚1217.170905166„ÃbïWÍÑ 2loss/SRqqt1/assignment_loss‚2.1696199098458657 /loss/FRt3/detection_loss‚1.6096987128257751 0loss/SRqqt4/detection_loss‚2.766214886019307 %loss/total_loss‚59.72556512257793 _runtime‚1217.170905166 0loss/FBt2/assignment_loss‚0.2140019008948002 _step‚83 2loss/SRqqt2/assignment_loss‚2.0325890015000883 2loss/SRqqt3/assignment_loss‚2.0481112748384476 0loss/FBt3/assignment_loss‚0.3105147234897034 /loss/FRt2/detection_loss‚2.4990482184945084 0loss/SRqqt2/detection_loss‚3.009302564289259 .loss/FBt3/detection_loss‚0.823603900270926 /loss/FRt2/assignment_loss‚5.671828525822337 /loss/FRt3/assignment_loss‚2.969424213233747 2loss/SRqqt4/assignment_loss‚2.4061553295581573 .loss/FRt4/detection_loss‚3.553209053842645 0loss/FBt1/assignment_loss‚0.5820515835657716 1loss/SRqqt3/detection_loss‚1.1478429412175166 % _timestamp‚1.7884651639854496e+09 /loss/FRt4/assignment_loss‚6.134607365256862 0loss/FBt4/assignment_loss‚0.5408336454597702 1loss/SRqqt1/detection_loss‚1.9060835144840753 /loss/FBt1/detection_loss‚3.6707089096307755 /loss/FBt2/detection_loss‚4.5162153244018555 .loss/FBt4/detection_loss‚2.685789752751589 trainer/global_step‚1049 /loss/FRt1/assignment_loss‚4.102292206552294 -loss/FRt1/detection_loss‚2.35581766433186 epoch‚0SJj¼‘΂(:† ˜ çÔžáÔ‡ gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚90.513' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚22.62825 gpu.0.smClock‚1200 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@¤žkaÏ‚(:Ö ˜ çÔþ”õ‡) disk./.usagePercent‚83.14326166968615 disk.sdb1.out‚0 disk.sda.in‚ 0.27734375" disk.nvme0n1.out‚ 7471.07421875 network.sent‚ 1.476631e+06 disk.md0p1.out‚ 128.18359375 disk.sdb.in‚ 0.15234375% disk./.usageGB‚2973.6057624816895 disk.md0.out‚ 128.18359375 disk.rbd2.out‚ 0.279296875 disk.dm-0.in‚ 477.57421875 disk.sda2.in‚ 0.27734375 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 0.279296875 disk.rbd2.in‚0" disk.nvme1n1.out‚ 8272.12109375 disk.sdb.out‚ 132.515625 disk.sda.out‚ 132.515625 disk.sda1.in‚0 proc.cpu.threads‚17 network.recv‚466207 disk.sdb2.in‚ 0.15234375 disk.sdb1.in‚0 disk.md0p1.in‚ 0.4296875 cpu‚24.82958639487584 disk.sdb2.out‚ 132.515625 disk.nvme1n1.in‚ 184.234375 disk.nvme0n1.in‚ 293.33984375% memory_percent‚36.843299865722656' proc.memory.availableMB‚ 10347.59375# proc.memory.rssMB‚ 2932.15234375* proc.memory.percent‚17.896437644958496 disk.sda2.out‚ 132.515625 disk.md0.in‚ 0.4296875 disk.dm-0.out‚ 15743.1953125 disk.sda1.out‚0  ¹Ð‚(:… § çÔ¿÷ˇ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚142.036' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.509 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M(¡vÑ‚(:ë § çÔðÈ°ˆ disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda.in‚ 0.27734375 disk.dm-0.out‚ 15808.8828125 disk.md0p1.in‚ 0.4296875 disk.nvme1n1.in‚ 184.234375" disk.nvme1n1.out‚ 8336.44921875 proc.cpu.threads‚17 disk.sdb.in‚ 0.15234375! disk.sdb2.out‚134.39697265625 disk.md0.in‚ 0.4296875$ memory_percent‚36.84346675872803# proc.memory.rssMB‚ 2932.15234375 network.sent‚ 1.513705e+06 disk.nvme0n1.in‚ 293.33984375 disk.rbd0.in‚0.015625* proc.memory.availableMB‚10347.56640625 disk.sdb.out‚134.39697265625" disk.nvme0n1.out‚ 7472.43359375 disk.sda1.out‚0 disk.sda2.in‚ 0.27734375! disk.sda2.out‚134.39697265625 disk.sdb2.in‚ 0.15234375 disk.rbd2.in‚0 network.recv‚469882 disk.md0.out‚ 130.0078125 disk.rbd2.out‚ 0.279296875* proc.memory.percent‚17.896437644958496 cpu‚24.831762574778697) disk./.usagePercent‚83.14002814899435 disk.sda.out‚134.39697265625 disk.dm-0.in‚ 477.57421875 disk.sda1.in‚0 disk.md0p1.out‚ 130.0078125 disk.rbd0.out‚ 0.279296875% disk./.usageGB‚2973.4901161193848Í_G’Ò‚(:‡ ¶ çÔª•‹‡ gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚134.575' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.64375 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹]º]wÓ‚(:ì ¶ çÔìõŒˆ proc.cpu.threads‚17 disk.sdb.out‚136.19677734375 disk.dm-0.out‚ 16036.3046875 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 0.279296875 disk.rbd2.out‚ 0.279296875# proc.memory.rssMB‚ 2932.15234375 disk.sdb1.out‚0" disk.nvme0n1.out‚ 7472.71484375" disk.nvme1n1.out‚ 8563.58984375 disk.dm-0.in‚ 477.57421875* proc.memory.availableMB‚10347.31640625 cpu‚24.834111318673127 disk.nvme0n1.in‚ 293.33984375 disk.sda1.out‚0 disk.sda2.in‚ 0.27734375 disk.md0p1.out‚ 131.73828125 disk.sda.in‚ 0.27734375 disk.nvme1n1.in‚ 184.234375 disk.sda.out‚136.19677734375 disk.md0p1.in‚ 0.4296875 disk.sdb2.in‚ 0.15234375 disk.md0.in‚ 0.4296875 disk.rbd2.in‚0 network.recv‚474471) disk./.usagePercent‚83.12989923822195$ disk./.usageGB‚2973.127857208252! disk.sda2.out‚136.19677734375 disk.md0.out‚ 131.73828125$ memory_percent‚36.84499263763428* proc.memory.percent‚17.896437644958496 network.sent‚ 1.519994e+06 disk.sdb.in‚ 0.15234375 disk.sdb1.in‚0 disk.sda1.in‚0! disk.sdb2.out‚136.19677734375=­ f2Ô- _wandbruntime‚1270  _runtime‚1270]—ZrÆÕÀ _runtime‚1270.785710989 _step‚84 & lr-AdamW/pg1‚0.0007494404629082335 & lr-AdamW/pg2‚0.0007494404629082335 trainer/global_step‚1099 % _timestamp‚1.7884652176005495e+098)~ÔŠÖÄ _step‚84 & lr-AdamW/pg1‚0.0007494404629082335 & lr-AdamW/pg2‚0.0007494404629082335 trainer/global_step‚1099 % _tim€Ñk @estamp‚1.7884652176005495e+09 _runtime‚1270.785710989TIÚ“,2×- _wandbruntime‚1270  _runtime‚1270ΑWÅ'Ø¿O OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚45869490454016 trainer/global_step‚1099 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49563329 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2159673549312 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2200070349824 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2153566307964 _step‚85 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚1270.786942975 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚41296999147600 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29423492 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11890920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚43232374497280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884652176017296e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19545404 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 313229312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚667 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2159906450944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚45392298951168 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29423492 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20114989 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336281088 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20114989 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11890918 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2199781060608 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚45392048046592 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29448338 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45392048046592 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2159673549312 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19546071 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚48069560803840 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚45392298951168 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2153795281500 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20114991 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2159906450944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20114991 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41296982108240 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚665 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7654486 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚48069247574528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 308438528 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 289289216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚43450548416204 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29448338 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚43232374497280 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7655151 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚43450794429100‡¢¨ÃÉ'ÙÃO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19545404 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 313229312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚667 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2159906450944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚45392298951168 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29423492 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20114989 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336281088 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20114989 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11890918 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2199781060608 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚45392048046592 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29448338 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45392048046592 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2159673549312 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19546071 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚48069560803840 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚45392298951168 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2153795281500 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20114991 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2159906450944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20114991 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41296982108240 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚665 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7654486 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚48069247574528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 308438528 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 289289216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚43450548416204 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29448338 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚43232374497280 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7655151 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚43450794429100 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚45869490454016 trainer/global_step‚1099 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49563329 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2159673549312 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2200070349824 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2153566307964 _step‚85 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚1270.786942975 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚41296999147600 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29423492 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11890920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚43232374497280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884652176017296e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532U%¿@2Ú- _wandbruntime‚1271  _runtime‚1271£]ÙÜÏ&ÛÉM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285081600 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29475056 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2161870490112 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2161637575168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚43490283275718 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49583448 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚45433549027328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚700 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚48112933681664 _step‚86 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7661812 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚43490037262614 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1099 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49608322 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48113242703360 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49608322 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚43271660534272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20133264 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29475056 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2155524796950 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚43271678537216 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45911150483968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20133264 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29450184 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20133266 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚45433549027328 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7661114 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2201783197696 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚41334512465664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2161637575168 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19563534 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11901720 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2155753770694 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19562834 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚45911174424064 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚41334529505024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚43271660534272 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚698 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11901722 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 309021696 _runtime‚1271.802087862š¢òôÎÜÍM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49583448 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚45433549027328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚700 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚48112933681664 _step‚86 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7661812 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚43490037262614 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1099 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49608322 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48113242703360 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49608322 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚43271660534272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20133264 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29475056 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2155524796950 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚43271678537216 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45911150483968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20133264 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29450184 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20133266 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚45433549027328 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7661114 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2201783197696 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚41334512465664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2161637575168 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19563534 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11901720 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2155753770694 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19562834 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚45911174424064 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚41334529505024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚43271660534272 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚698 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11901722 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 309021696 _runtime‚1271.802087862 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285081600 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29475056 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2161870490112 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2161637575168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚43490283275718VàäÕZ2Ý- _wandbruntime‚1271  _runtime‚1271öðMXÞÒ 1loss/SRqqt3/detection_loss‚1.0730466214962393 1loss/SRqqt4/detection_loss‚3.3971741408109666 /loss/FBt1/detection_loss‚2.1598530411720276 /loss/FRt4/assignment_loss‚5.393826394245543 % _timestamp‚1.7884652186190972e+09 0loss/FRt1/assignment_loss‚3.8090772497944716 2loss/SRqqt3/assignment_loss‚1.9950206510453339 0loss/FBt1/assignment_loss‚0.7436031136247847 1loss/FBt3/assignment_loss‚0.37619254190796597 /loss/FRt3/detection_loss‚1.6769211888313293 1loss/SRqqt1/detection_loss‚1.9713057724178815 /loss/FBt3/detection_loss‚0.7123157258521132 .loss/FRt2/detection_loss‚2.736730245443491 /loss/FRt4/detection_loss‚3.0493999308553237 .loss/FBt2/detection_loss‚3.355186777455466 .loss/FBt4/detection_loss‚3.399135481227528 _runtime‚ 1271.80380475 _step‚87 1loss/SRqqt1/assignment_loss‚1.901707819739326 2loss/SRqqt4/assignment_loss‚2.7192650072276594 0loss/FBt4/assignment_loss‚0.6746554295109077 /loss/FRt2/assignment_loss‚5.041841109593709 /loss/FRt1/detection_loss‚2.4394002600413995 trainer/global_step‚1099 epoch‚0 0loss/FRt3/assignment_loss‚2.2473299720930675 2loss/SRqqt2/assignment_loss‚2.4636814142427137 0loss/FBt2/assignment_loss‚1.0303803898942923 1loss/SRqqt2/detection_loss‚2.7387419285312777 %loss/total_loss‚57.10579220705482JÌZ¬\ßÖ 1loss/SRqqt1/assignment_loss‚1.901707819739326 2loss/SRqqt4/assignment_loss‚2.7192650072276594 0loss/FBt4/assignment_loss‚0.6746554295109077 /loss/FRt2/assignment_loss‚5.041841109593709 /loss/FRt1/detection_loss‚2.4394002600413995 trainer/global_step‚1099 epoch‚0 0loss/FRt3/assignment_loss‚2.2473299720930675 2loss/SRqqt2/assignment_loss‚2.4636814142427137 0loss/FBt2/assignment_loss‚1.0303803898942923 1loss/SRqqt2/detection_loss‚2.7387419285312777 %loss/total_loss‚57.10579220705482 1loss/SRqqt3/detection_loss‚1.0730466214962393 1loss/SRqqt4/detection_loss‚3.3971741408109666 /loss/FBt1/detection_loss‚2.1598530411720276 /loss/FRt4/assignment_loss‚5.393826394245543 % _timestamp‚1.7884652186190972e+09 0loss/FRt1/assignment_loss‚3.8090772497944716 2loss/SRqqt3/assignment_loss‚1.9950206510453339 0loss/FBt1/assignment_loss‚0.7436031136247847 1loss/FBt3/assignment_loss‚0.37619254190796597 /loss/FRt3/detection_loss‚1.6769211888313293 1loss/SRqqt1/detection_loss‚1.9713057724178815 /loss/FBt3/detection_loss‚0.7123157258521132 .loss/FRt2/detection_loss‚2.736730245443491 /loss/FRt4/detection_loss‚3.0493999308553237 .loss/FBt2/detection_loss‚3.355186777455466 .loss/FBt4/detection_loss‚3.399135481227528 _runtime‚ 1271.80380475 _step‚87W£9pà‚(:… Å çÔ› …ˆ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚134.908' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.727 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0T@aná‚(:ã Å çÔ¤ØÈ disk.rbd2.out‚ 0.279296875 disk.nvme1n1.out‚ 8698.984375 disk.sda2.in‚ 0.27734375 disk.sdb2.in‚ 0.15234375$ disk./.usageGB‚2973.127212524414 disk.sdb.in‚ 0.15234375 disk.nvme0n1.in‚ 293.33984375 disk.nvme0n1.out‚ 7472.765625 disk.md0.in‚ 0.4296875 disk.rbd0.in‚0.015625$ memory_percent‚36.84365749359131* proc.memory.percent‚17.896437644958496 network.recv‚478798 disk.sdb1.in‚0 disk.dm-0.out‚16171.75 disk.md0.out‚ 133.09765625# proc.memory.rssMB‚ 2932.15234375 network.sent‚ 1.523595e+06) disk./.usagePercent‚83.12988121259201 disk.sdb1.out‚0 disk.md0p1.in‚ 0.4296875! disk.sdb2.out‚137.61767578125 disk.rbd0.out‚ 0.279296875 cpu‚24.836187294475685 disk.nvme1n1.in‚ 184.234375 disk.dm-0.in‚ 477.57421875 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0p1.out‚ 133.09765625 disk.sdb.out‚137.61767578125 disk.sda.in‚ 0.27734375 disk.sda.out‚137.61767578125! disk.sda2.out‚137.61767578125* proc.memory.availableMB‚10347.53515625 proc.cpu.threads‚17 disk.rbd2.in‚0ÿœâ‚(:‘ Ô çÔ¶êˇ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñóÄtã‚(:é Ô çÔ¯·žˆ# proc.memory.rssMB‚ 2932.15234375 network.sent‚ 1.575176e+06 network.recv‚484461) disk./.usagePercent‚83.13146416154925 disk.md0p1.in‚ 0.4296875! disk.sdb2.out‚139.56982421875 disk.sda.out‚139.56982421875 disk.nvme1n1.in‚ 184.234375 disk.nvme0n1.in‚ 293.33984375 disk.dm-0.out‚ 16178.703125 disk.md0p1.out‚ 134.98046875 disk.rbd0.in‚0.015625 disk.sdb1.in‚0 disk.sda.in‚ 0.27734375 disk.sda1.out‚0* proc.memory.percent‚17.896437644958496 proc.cpu.threads‚17 disk.sdb.out‚139.56982421875 disk.sda2.in‚ 0.27734375$ memory_percent‚36.84365749359131 cpu‚24.8384331614678 disk.rbd2.in‚0 disk.rbd2.out‚ 0.279296875$ disk./.usageGB‚2973.183826446533 disk.md0.out‚ 134.98046875" disk.nvme1n1.out‚ 8705.42578125 disk.sdb.in‚ 0.15234375! disk.sda2.out‚139.56982421875* proc.memory.availableMB‚10347.53515625 disk.sdb2.in‚ 0.15234375 disk.md0.in‚ 0.4296875 disk.rbd0.out‚ 0.279296875 disk.sdb1.out‚0" disk.nvme0n1.out‚ 7473.27734375 disk.dm-0.in‚ 477.57421875 disk.sda1.in‚0Qk¶’ä‚(:‡ ã çÔ¹€Ê‡ gpu.0.gpu‚26.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚116.583' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.14575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03xÚså‚(:è ã çÔ¡¶®ˆ disk.sdb2.in‚ 0.15234375 disk.nvme1n1.in‚ 184.234375 disk.sda2.in‚ 0.27734375 disk.md0p1.out‚ 136.40234375) disk./.usagePercent‚83.13438004717638$ disk./.usageGB‚2973.288112640381 disk.sdb.out‚141.05615234375! disk.nvme0n1.out‚ 7554.1640625# proc.memory.rssMB‚ 2932.15234375 proc.cpu.threads‚17 network.sent‚ 1.580458e+06 disk.rbd2.in‚0 disk.sdb.in‚ 0.15234375 disk.sda.in‚ 0.27734375 disk.sda.out‚141.05615234375 disk.md0p1.in‚ 0.4296875$ memory_percent‚36.84394359588623 disk.md0.out‚ 136.40234375 disk.rbd0.in‚0.015625 disk.nvme0n1.in‚ 293.33984375 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.availableMB‚10347.48828125* proc.memory.percent‚17.896437644958496 network.recv‚488471! disk.sdb2.out‚141.05615234375 disk.md0.in‚ 0.4296875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.dm-0.in‚ 477.57421875 disk.rbd2.out‚ 0.279296875! disk.nvme1n1.out‚ 8772.0390625 cpu‚24.84079966810517 disk.rbd0.out‚ 0.279296875 disk.dm-0.out‚ 16326.203125! disk.sda2.out‚141.05615234375t‡S/’æ‚(:‡ ò çÔ‘€ß‡ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚138.919' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.72975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»´y1hç‚(:Ý ò çÔ‡–ˆ disk.rbd2.out‚ 0.279296875 disk.sda1.out‚0* proc.memory.percent‚17.896437644958496 proc.cpu.threads‚17% disk./.usageGB‚2973.4758911132812 disk.md0.out‚ 137.859375 disk.sdb.in‚ 0.15234375 disk.nvme0n1.in‚ 293.33984375 disk.sda2.in‚ 0.27734375* proc.memory.availableMB‚10347.48046875! disk.sdb2.out‚142.58349609375 disk.md0.in‚ 0.4296875 disk.rbd0.out‚ 0.279296875 disk.nvme1n1.in‚ 184.234375 disk.nvme1n1.out‚8880.75 disk.sda.in‚ 0.27734375 disk.dm-0.in‚ 477.57421875 disk.md0p1.out‚ 137.859375 network.sent‚ 1.586791e+06 network.recv‚492938 disk.sdb.out‚142.58349609375 cpu‚24.84275599844069) disk./.usagePercent‚83.13963041187007 disk.sda.out‚142.58349609375 disk.dm-0.out‚ 16467.15625 disk.sda1.in‚0$ memory_percent‚36.84399127960205! disk.sda2.out‚142.58349609375 disk.sdb2.in‚ 0.15234375 disk.rbd2.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.nvme0n1.out‚ 7586.40625 disk.md0p1.in‚ 0.4296875# proc.memory.rssMB‚ 2932.15234375 disk.rbd0.in‚0.015625ÍZ1|2è- _wandbruntime‚1327  _runtime‚1327¢Ã\%ÆéÀ & lr-AdamW/pg1‚0.0007835369352880439 & lr-AdamW/pg2‚0.0007835369352880439 trainer/global_step‚1149 % _timestamp‚1.7884652742189128e+09 _runtime‚1327.403855677 _step‚88“®î3ÊêÄ & lr-AdamW/pg1‚0.0007835369352880439 & lr-AdamW/pg2‚0.0007835369352880439 trainer/global_step‚1149 % _timestamp‚1.7884652742189128e+09 _runtime‚1327.403855677 _step‚88X¹-¨62ë- _wandbruntime‚1327  _runtime‚13276'âÅ'ì¿O TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21028741 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚45196676346880 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2251504348472 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚45196694349824 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2258122157568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30786290 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2251733322008 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2257889255936 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21028739 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚701 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8006420 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚45425004327912 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚45196694349824 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30761444 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30761444 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚47454816507392 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51790183 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2258122157568 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚50253405171200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21028741 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336281088 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚703 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 291386368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚47454565602816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12431018 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 315326464 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚47454565602816 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20437440 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8005719 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚47953665016320 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 trainer/global_step‚1149 _runtime‚1327.404945022 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚47454565602816 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20437440 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8005719 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚47953665016320 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 trainer/global_step‚1149 _runtime‚1327.404945022 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚45196676346880 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2251504348472 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚45196694349824 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2258122157568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30786290 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2251733322008 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2257889255936 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21028739 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚701 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8006420 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚45425004327912 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚45196694349824Y•Ù#2î- _wandbruntime‚1328  _runtime‚1328ÿHyMÏ&ïÉM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20454572 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12441820 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚689 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21047016 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2259853281792 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚47995372926464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚1149 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51860024 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2253462837458 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21047014 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚45235962383872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚43211047376224 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30788136 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚45235980386816 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30813008 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 289275904 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2301733713920 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12441822 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚45464493174322 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43211030336864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚50297082700288 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51860024 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2253691811202 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚47995348986368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 % _timestamp‚1.7884652753559716e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚47495815665664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30788136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚1328.541548247 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51835150 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2259853281792 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8012752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0AØ(Ó&ðÍM OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20455263 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚50297395916288 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 313216000 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚45235980386816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚45464739187426 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚90 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚47496066583552 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30813008 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21047014 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8012752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20454572 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12441820 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚689 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21047016 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2259853281792 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚47995372926464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚1149 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51860024 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2253462837458 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21047014 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚45235962383872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚43211047376224 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30788136 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚45235980386816 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30813008 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 289275904 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2301733713920 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12441822 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚45464493174322 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43211030336864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚50297082700288 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51860024 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2253691811202 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚47995348986368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 % _timestamp‚1.7884652753559716e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚47495815665664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30788136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚1328.541548247 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51835150 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2259853281792 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2356337864192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚49517334063616 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336281088 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2399848451072 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21942491 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚47399460239620 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚1385.336614111 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚47160978196480 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32099396 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _step‚93 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54066733 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚49517083159040 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚50037863518720 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚49517334063616 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚47160978196480 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚49517083159040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21942491 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12971120 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚50037887458816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚45050017850640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32099396 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚45050034890000 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21326502 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚47399706252516 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21942489 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2356337864192 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8355384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32124242 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2349671362516 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12971118 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 308438528 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 trainer/global_step‚1199 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚47160996199424 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚52437711969792 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2349442388980 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2356337864192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚49517334063616 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336281088 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2399848451072 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21942491 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚47399460239620 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚1385.336614111 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚47160978196480 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32099396 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _step‚93 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54066733 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚49517083159040 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚50037863518720 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚49517334063616 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚47160978196480 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚49517083159040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21942491 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12971120 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚50037887458816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚45050017850640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32099396 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚45050034890000 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21326502 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚47399706252516 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21942489 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2356337864192 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8355384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32124242 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2349671362516 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36] t’2‚- _wandbruntime‚1386  _runtime‚1386ý¾fÏ&ƒÉM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2358301903360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54086852 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 % _timestamp‚1.7884653333432958e+09 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚50079571428864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚47200264233472 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54111726 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚732 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚52481382140416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 299761664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 trainer/global_step‚1199 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32126088 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚47200264233472 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21960766 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12981922 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336281088 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21960766 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21344914 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚49558584139776 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2358068988416 _runtime‚1386.530185718 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12981920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2401834651648 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚47439195099134 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚49558333221888 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21960764 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21344182 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273?X6™#„ÍM JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚730 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21960764 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚47438949086030 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚47200282236416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 HBDeviceStatsMonitorºeÖ°#.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚50079547488768 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚47200264233472 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54111726 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚732 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚52481382140416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 299761664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 trainer/global_step‚1199 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32126088 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚47200264233472 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21960766 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12981922 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336281088 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21960766 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21344914 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚49558584139776 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2358068988416 _runtime‚1386.530185718 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12981920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2401834651648 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚47439195099134 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚49558333221888 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21960764 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21344182 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2358301903360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54086852 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 % _timestamp‚1.7884653333432958e+09 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚50079571428864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ODeviceStatsMonitor.on_validation_batch_start/segment.all.freed"trainer/global_step2:JPï:ö,wÇbr ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JPŽN*teÈb` =DeviceStatsMonitor.on_validation_batch_start/segment.all.peak"trainer/global_step2:JP½[uÉbp MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JP*Ý.tÊbo LDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JP9Å ðyËbt QDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JP7“DgÌbb ?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections"trainer/global_step2:JPó³jÂkÍbf CDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak"trainer/global_step2:JP^™¸¢vÎbq NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JP'M‚-yÏbt QDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JP=Д©yÐbt QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP4Uƒ‚lÑbg DDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak"trainer/global_step2:JP­9gvÒbq NDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JPö/wÓbr ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JPÓ-ö6kÔbf CDeviceStatsMonitor.on_validation_batch_start/allocation.all.current"trainer/global_step2:JP>îhÕbc @DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak"trainer/global_step2:JP!þrupÖbk HDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed"trainer/global_step2:JPìð{n×bi FDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current"trainer/global_step2:JPš¬˜…kØbf CDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak"trainer/global_step2:JPVó{jÙbe BDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak"trainer/global_step2:JPjTuÚbp MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JPxŽT¢fÛba >DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries"trainer/global_step2:JP’V(sÜbn KDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated"trainer/global_step2:JPLÚœÎsÝbn KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JPJ¤PiÞbd ADeviceStatsMonitor.on_validation_batch_start/active.all.allocated"trainer/global_step2:JPã„Çkßbf CDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed"trainer/global_step2:JP) H~àby VDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPø`õŸtábo LDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JP(žsâbn KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JP0DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚49027056604160 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚49027074607104 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6561988608 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 2399212239872 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚309 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚49027074607104 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 175409664 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚19927 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚30751 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚22878957 trainer/global_step‚1208 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚19925 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚30508 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6966133760 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 2399036830208 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 572522496 NADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚55706592 VCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚51426093434368 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 2443393512960 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 564258816 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚32807708 \PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚8697400 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚49027056604160 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚49027074607104 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6561988608 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 2399212239872 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚309 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚49027074607104 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 175409664 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚19927 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚30751 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚22878957 trainer/global_step‚1208 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚19925 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚30508 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6966133760 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 2399036830208 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 572522496 NADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚55706592 VCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚51426093434368 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 2443393512960 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 564258816 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚32807708 \PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚8697400 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current"trainer/global_step2:JPØŠV|jªbe BDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak"trainer/global_step2:JP~{ÜLe«b` =DeviceStatsMonitor.on_validation_batch_end/active.all.current"trainer/global_step2:JPC¡iVw¬br ODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JP1Lt„s­bn KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JP÷T’xk®bf CDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed"trainer/global_step2:JP¦PŒv¯bq NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JP ¥”7j°be BDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed"trainer/global_step2:JPõAHn±bi FDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated"trainer/global_step2:JPZZj²be BDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed"trainer/global_step2:JP)RR¢t³bo LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JPrÒdCr´bm JDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JPïÁxµbs PDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPßç¿k¶bf CDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed"trainer/global_step2:JP£~Él·bg DDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current"trainer/global_step2:JP‡üi‰c¸b^ ;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc"trainer/global_step2:JPë~Dor¹bm JDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JP¥ažÆcºb^ ;DeviceStatsMonitor.on_validation_batch_end/active.all.freed"trainer/global_step2:JP<\t÷k»bf CDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current"trainer/global_step2:JPNŽF$i¼bd ADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed"trainer/global_step2:JP”ÊëÅd½b_ DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak"trainer/global_step2:JPnM¦tãbo LDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current"trainer/global_step2:JPl*öe[äbV 3DeviceStatsMonitor.on_validation_batch_end/num_ooms"trainer/global_step2:JP¤6øNråbm JDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JP˜–Czgæbb ?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams"trainer/global_step2:JP¡§œòlçbg DDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current"trainer/global_step2:JP#'ppèbk HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated"trainer/global_step2:JPQþósébn KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JP×Ä8(oêbj GDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JPdsëbn KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JPÑô¾Ñmìbh EDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak"trainer/global_step2:JP¢ŽÚgíbb ?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed"trainer/global_step2:JPÇ>oîbj GDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current"trainer/global_step2:JPJ4>$nïbi FDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed"trainer/global_step2:JPGÊkðbf CDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak"trainer/global_step2:JPÛ5•uñbp MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current"trainer/global_step2:JP7@S qòbl IDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JPÆ.¾Kdób_ DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚273 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚19927 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚49057140281344 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 2399510690304 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚22365814 ZNDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚8702975 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚32836541 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚32816616 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚49057140281344 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 564258816 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 2443721918464 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788465405816216e+09 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚582 trainer/global_step‚1208 _runtime‚1459.001558065 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚13663238 H;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚55710955 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 193412608 TADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚51456632968704 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚30751 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 VJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚8702576 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 [HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚49096203603452 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚46703303906480 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 564258816 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚49057122278400 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚30508 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚55710955 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0Æ}’y9)³R RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6741051909 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 572522496 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚273 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚19927 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚49057140281344 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 2399510690304 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚22365814 ZNDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚8702975 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚32836541 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚32816616 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚49057140281344 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 564258816 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 2443721918464 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788465405816216e+09 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚582 trainer/global_step‚1208 _runtime‚1459.001558065 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚13663238 H;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚55710955 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 193412608 TADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚51456632968704 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚30751 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 VJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚8702576 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 [HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚49096203603452 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚46703303906480 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 564258816 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚49057122278400 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚30508 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚55710955 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚22894339aŽ_äÕ‘Ž‚(:† ˆ¢çÔæÚˆˆ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+ ËZv‚(:ë ˆ¢çÔð•Úˆ# proc.memory.rssMB‚ 3033.70703125 cpu‚25.034031684523196 network.sent‚ 1.741722e+06 network.recv‚535653 disk.nvme0n1.in‚ 330.71875 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0.015625! disk.sda2.out‚158.73876953125 disk.md0.in‚ 0.4296875! disk.nvme1n1.out‚ 10172.515625 disk.dm-0.in‚ 590.86328125 disk.sdb1.in‚0) proc.memory.percent‚18.51627826690674) disk./.usagePercent‚83.17284514153597 disk.sda2.in‚ 0.27734375 disk.md0p1.out‚ 153.42578125 disk.rbd0.out‚ 0.279296875 disk.rbd2.in‚0* proc.memory.availableMB‚10248.08203125 proc.cpu.threads‚17% disk./.usageGB‚2974.6638107299805! disk.sdb2.out‚158.73876953125 disk.sda.in‚ 0.27734375 disk.sda.out‚158.73876953125 disk.md0p1.in‚ 0.4296875 disk.sdb2.in‚ 0.15234375 disk.md0.out‚ 153.42578125 disk.nvme1n1.in‚ 260.14453125" disk.nvme0n1.out‚ 8645.50390625 disk.dm-0.out‚18818.01953125$ memory_percent‚37.45067119598389 disk.rbd2.out‚ 0.279296875 disk.sdb.in‚ 0.15234375 disk.sdb.out‚158.73876953125 disk.sdb1.out‚0ç1ýÄJbE "Purity/2FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÛm‡J‘bE "Purity/3FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPsÌ¢¤J’bE "Purity/1FRt*SRqqt4FBt/event_purity"trainer/global_step2:JP6GN“bI &Purity/*FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP»(¼ÑH”bC Purity/*FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPMÛ|N•bI &Purity/0FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP0CÛ¯N–bI &Purity/1FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPÎÏüaN—bI &Purity/3FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPÔB;˜b6 jet/accuracy_4_of_4"trainer/global_step2:JP‹~N–H™bC Purity/3FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPfË=@šb; particle/accuracy_6_of_7"trainer/global_step2:JP‹Ž´™J›bE "Purity/*FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP[wšÒHœbC Purity/*FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JP”JÿHbC Purity/1FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPî4áJžbE "Purity/2FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP¥ÍHHŸbC Purity/2FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP9Íè3@ b; particle/accuracy_5_of_6"trainer/global_step2:JPmoµJ¡bE "Purity/3FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP‚›J¢bE "Purity/*FRt2SRqqt3FBt/event_purity"trainer/global_step2:JP„s|mJ£bE "Purity/4FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP,bJ^H¤bC Purity/3FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPu>œJ¥bE "Purity/*FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÌ„a®H¦bC Purity/1FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPu™ÁN§bI &Purity/2FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JP ÇI÷H¨bC Purity/3FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP¾q7J©bE "Purity/2FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP•u$5NªbI &Purity/*FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPf]ÅÊN«bI &Purity/2FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPþ ãJ¬bE "Purity/3FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPŽ·Ýn;­b6 jet/accuracy_3_of_4"trainer/global_step2:JPÂا;®b6 jet/accuracy_4_of_5"trainer/global_step2:JP´ÛÆ&H¯bC Purity/0FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JPñþ”H°bC Purity/*FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP`p"N±bI &Purity/*FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPY›ÎìJ²bE "Purity/0FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP¯€ÏJ³bE "Purity/0FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPMntçN´bI &Purity/0FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP!Á6`NµbI &Purity/1FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP-§3N¶bI &Purity/3FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP÷*+@·b; particle/accuracy_2_of_4"trainer/global_step2:JPÁ¸+H¸bC Purity/*FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPrïõ¢N¹bI &Purity/2FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP\ÍEHºbC Purity/2FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPÊÎ$J»bE "Purity/*FRt3SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP\úž¢J¼bE "Purity/0FRt2SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPTŠº`H½bC Purity/*FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPKMvfJ¾bE "Purity/*FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP!ªìqN¿bI &Purity/3FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPÊoðNÀbI &Purity/3FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPéÐj@Áb; particle/accuracy_2_of_6"trainer/global_step2:JPÄ?ßÌJÂbE "Purity/*FRt2SRqqt2FBt/event_purity"trainer/global_step2:JP¬dNüJÃbE "Purity/*FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPµkžJÄbE "Purity/*FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÍŸ)ÍHÅbC Purity/1FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPÆB mNÆbI &Purity/2FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP˜B¿JÇbE "Purity/2FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPsƒG1HÈbC Purity/3FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPÝ.JÉbE "Purity/2FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÜëeHÊbC Purity/1FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JPJýÑšNËbI &Purity/*FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPc/<ÒJÌbE "Purity/*FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP†?k1NÍbI &Purity/2FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPsF¥*NÎbI &Purity/4FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPV:ß@Ïb; particle/accuracy_1_of_3"trainer/global_step2:JPì à HÐbC Purity/1FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JPŸ):êJÑbE "Purity/*FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPÍ¿ªJÒbE "Purity/*FRt4SRqqt1FBt/event_purity"trainer/global_step2:JP®´÷NÓbI &Purity/*FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP`{:ŠNÔbI &Purity/*FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP4|G¢JÕbE "Purity/1FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPSµñJÖbE "Purity/1FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP(Õ žN×bI &Purity/3FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPÓ‘ËJØbE "Purity/3FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPB-Ñ‹JÙbE "Purity/0FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPŸî\@Úb; particle/accuracy_2_of_7"trainer/global_step2:JPÞÆø$JÛbE "Purity/*FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP“*&HÜbC Purity/*FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JPN¿?JÝbE "Purity/*FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPSÌ”QHÞbC Purity/2FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JP¦g ÌNßbI &Purity/0FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPßX¿ÇHàbC Purity/1FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP߈ŒHábC Purity/2FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP+DÁNâbI &Purity/4FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP@âÞlJãbE "Purity/3FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPçrßNäbI &Purity/1FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPÑ85ÊNåbI &Purity/2FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP’;رHæbC Purity/3FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPV«©_JçbE "Purity/0FRt*SRqqt4FBt/event_purity"trainer/global_step2:JP7¼gqJèbE "Purity/*FRt1SRqqt3FBt/event_purity"trainer/global_step2:JPâ—JébE "Purity/0FRt0SRqqt4FBt/event_purity"trainer/global_step2:JP–` —NêbI &Purity/2FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP¿ÛHNëbI &Purity/2FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPÃ2)¡NìbI &Purity/3FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP° 5eNíbI &Purity/4FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP´€InJîbE "Purity/*FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPuÛ1¢;ïb6 jet/accuracy_4_of_7"trainer/global_step2:JPàµpJðbE "Purity/0FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPn¨¢ÞNñbI &Purity/0FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPÈãHòbC Purity/2FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPÒó JóbE "Purity/1FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPW‡ÿÑHôbC Purity/0FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPÁ`ÍdNõbI &Purity/1FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPÈÊHöbC Purity/2FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP£7 ~J÷bE "Purity/3FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP7<ŸU;øb6 jet/accuracy_1_of_7"trainer/global_step2:JP_/ÃÌ;ùb6 jet/accuracy_2_of_7"trainer/global_step2:JP{8JúbE "Purity/1FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP\>¿¿NûbI &Purity/0FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPœœNübI &Purity/1FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPVÛzÙNýbI &Purity/1FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPš º@NþbI &Purity/2FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP×ïß”HÿbC Purity/2FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP?eâÌH€ bC Purity/1FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JP7×!J bE "Purity/0FRt1SRqqt3FBt/event_purity"trainer/global_step2:JP^W}”H‚ bC Purity/*FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JP[(ý/Hƒ bC Purity/*FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPæFjûN„ bI &Purity/*FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPó¼ó@… b; particle/accuracy_3_of_3"trainer/global_step2:JP]xö¨J† bE "Purity/3FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPäkÀ¬H‡ bC Purity/3FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP¶Ú½Jˆ bE "Purity/1FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPí,`„J‰ bE "Purity/2FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPyÚ‘ÃHŠ bC Purity/1FRt*SRqqt4FBt/FRt_purity"trainer/global_step2:JPR¨€J‹ bE "Purity/0FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPO†:NŒ bI &Purity/0FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPN bI &Purity/2FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP¡`xJŽ bE "Purity/3FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPXœ#DJ bE "Purity/*FRt*SRqqt4FBt/event_purity"trainer/global_step2:JP¯â ; b6 jet/accuracy_3_of_7"trainer/global_step2:JP|ýp·H‘ bC Purity/2FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JP\Ñ´J’ bE "Purity/3FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPjƒ†J“ bE "Purity/1FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP{ DêN” bI &Purity/1FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPWÚ.J• bE "Purity/1FRt3SRqqt0FBt/event_purity"trainer/global_step2:JP#³³H– bC Purity/2FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP©ÉN— bI &Purity/3FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP^ß@˜ b; particle/accuracy_1_of_2"trainer/global_step2:JPîF“õH™ bC Purity/3FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP;¹"Jš bE "Purity/3FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPѾ܋H› bC Purity/*FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPP¹· Nœ bI &Purity/4FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPÑT ; b6 jet/accuracy_6_of_6"trainer/global_step2:JPÿOEôJž bE "Purity/0FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPjYÐñJŸ bE "Purity/1FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPÚFhD;  b6 loss/val_total_loss"trainer/global_step2:JPžï7€J¡ bE "Purity/*FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPc*8xJ¢ bE "Purity/*FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP`ÿ¥N£ bI &Purity/0FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPÁMmÔH¤ bC Purity/0FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPü§ÛH¥ bC Purity/2FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP?aŸN¦ bI &Purity/3FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP HbþH§ bC Purity/3FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPÞÐSýN¨ bI &Purity/*FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP™/ÏÒH© bC Purity/1FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPûº”Nª bI &Purity/3FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPvh ÈNó bI &Purity/1FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPŽ²éJô bE "Purity/2FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÇoyôHõ bC Purity/2FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPÈ#IûNö bI &Purity/3FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPÚ㯦N÷ bI &Purity/3FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPO05Jø bE "Purity/4FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP®fI]Nù bI &Purity/*FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPÿT¦Jú bE "Purity/0FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPJû bE "Purity/2FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPD/bNü bI &Purity/3FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP+1ÝðHý bC Purity/*FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JP¾í:Jþ bE "Purity/3FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPö›|[Hÿ bC Purity/0FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JP‚1(ÍN€ bI &Purity/*FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP§ÑÓ@ b; particle/accuracy_3_of_5"trainer/global_step2:JPžd5J‚ bE "Purity/1FRt2SRqqt2FBt/event_purity"trainer/global_step2:JP½6äHƒ bC Purity/*FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP†{[N„ bI &Purity/0FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPÿ] J… bE "Purity/2FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPê6_J† bE "Purity/2FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¨ý RJ‡ bE "Purity/2FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP*ÔEHˆ bC Purity/2FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JPKh•H‰ bC Purity/*FRt3SRqqt2FBt/FRt_purity"trainer/global_step2:JP«­ððJŠ bE "Purity/*FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPá°H‹ bC Purity/*FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPÖÞ¢HŒ bC Purity/2FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPa #NH bC Purity/*FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP‚qüNŽ bI &Purity/*FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPýˆŸJ bE "Purity/1FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPÎæÃòH bC Purity/2FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP¶¿$@‘ b; particle/accuracy_6_of_6"trainer/global_step2:JPe¿ÁH’ bC Purity/1FRt3SRqqt2FBt/FRt_purity"trainer/global_step2:JPˆm‘`J“ bE "Purity/*FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP±:•^J” bE "Purity/0FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPL%D?N• bI &Purity/2FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP.ü(J– bE "Purity/3FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPäÙ©FH— bC Purity/0FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JPÒ±D;˜ b6 jet/accuracy_1_of_3"trainer/global_step2:JP´‰›N™ bI &Purity/*FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP/ù§Nš bI &Purity/1FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JP}23N› bI &Purity/2FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPèÞ‘Nœ bI &Purity/4FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP]Å3ºJ bE "Purity/0FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPS8aHž bC Purity/0FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JPwªâ§HŸ bC Purity/*FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP›‡wBH  bC Purity/*FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPô†§èH¡ bC Purity/2FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPïV…¤N¢ bI &Purity/2FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPaHeJ£ bE "Purity/0FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP4EÇJ¤ bE "Purity/0FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPsKÜvH¥ bC Purity/1FRt0SRqqt4FBt/FRt_purity"trainer/global_step2:JP!ïhÎN¦ bI &Purity/1FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP‚dkJ§ bE "Purity/2FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP9›bêJ¨ bE "Purity/*FRt0SRqqt4FBt/event_purity"trainer/global_step2:JPNˆ‡J© bE "Purity/0FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP}¿ÛNª bI &Purity/0FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPTÑJ¯ bE "Purity/0FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP4$œêJ° bE "Purity/*FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP—ÚÕ„H± bC Purity/1FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPÅÉñJ² bE "Purity/1FRt*SRqqt3FBt/event_purity"trainer/global_step2:JP¶¢€J³ bE "Purity/2FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPúzá6N´ bI &Purity/2FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPù mJµ bE "Purity/3FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPÌâ™J¶ bE "Purity/*FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP¾ÄæN· bI &Purity/1FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPÈÊH©N¸ bI &Purity/3FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP=Ò“H¹ bC Purity/0FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPIm"=Nº bI &Purity/0FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPKYgN» bI &Purity/4FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPHLÓ;¼ b6 jet/accuracy_2_of_3"trainer/global_step2:JPù#º€J½ bE "Purity/*FRt3SRqqt2FBt/event_purity"trainer/global_step2:JP 1"ÄN¾ bI &Purity/*FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPi×—ÉN¿ bI &Purity/0FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPíÛ}rNÀ bI &Purity/2FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP eÎ'NÁ bI &Purity/4FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPd. N bI &Purity/4FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP\uLƒ;à b6 jet/accuracy_2_of_5"trainer/global_step2:JP÷ÒuhJÄ bE "Purity/*FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPæ‰&HÅ bC Purity/0FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP[¡OÓHÆ bC Purity/1FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPCxJÇ bE "Purity/*FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPAËâNÈ bI &Purity/*FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPç˜~JÉ bE "Purity/0FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPZÆnØJÊ bE "Purity/0FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPœ†‘õHË bC Purity/1FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPöÃÝvNÌ bI &Purity/2FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP¤g[«NÍ bI &Purity/4FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPn ôJÎ bE "Purity/*FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPÄŒÛ(NÏ bI &Purity/2FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP× Ù@Ð b; particle/accuracy_5_of_5"trainer/global_step2:JPM½ èJÑ bE "Purity/2FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPyZY–HÒ bC Purity/1FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JP¨ßŒHÓ bC Purity/1FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPðt^ HÔ bC Purity/2FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP|º¿NÕ bI &Purity/2FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPP`s+JÖ bE "Purity/3FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPeøû;× b6 jet/accuracy_2_of_2"trainer/global_step2:JPÂO&JØ bE "Purity/3FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPsüÍHÙ bC Purity/1FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JP„%1ÏNÚ bI &Purity/0FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPAhoaJÛ bE "Purity/0FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPõ-˜NÜ bI &Purity/1FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPs¥ñ^NÝ bI &Purity/1FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP_6Ë¡HÞ bC Purity/2FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP>?íJß bE "Purity/2FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP%kµHà bC Purity/3FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP, AJá bE "Purity/0FRt2SRqqt3FBt/event_purity"trainer/global_step2:JPvÒI]Jâ bE "Purity/*FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPòMoHã bC Purity/*FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPâ 4ˆJä bE "Purity/0FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPöÿ'HNå bI &Purity/3FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP1qcJæ bE "Purity/3FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP°LgÎNç bI &Purity/3FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP%7;è b6 jet/accuracy_1_of_6"trainer/global_step2:JPl’°_Jé bE "Purity/4FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPÏ-›Jê bE "Purity/4FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPC6ÌgHë bC Purity/*FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP&ó,Nì bI &Purity/0FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPÌE­Hí bC Purity/1FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP¶™zhHî bC Purity/2FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPïK¶Nï bI &Purity/2FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP\ÛsžJð bE "Purity/2FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPib}eJñ bE "Purity/0FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPLZ%Jò bE "Purity/4FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPðZö>Nó bI &Purity/0FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPë:m"Nô bI &Purity/3FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPQû~ÏJõ bE "Purity/*FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP²ü¯{Jö bE "Purity/1FRt1SRqqt3FBt/event_purity"trainer/global_step2:JPK|¦N÷ bI &Purity/0FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPrcîJø bE "Purity/1FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPH¤‰Jù bE "Purity/1FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPîå›ìNú bI &Purity/4FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP…ö†Hû bC Purity/3FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP#ò4óHü bC Purity/2FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JPƪcNJý bE "Purity/3FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPÓJþ bE "Purity/4FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP妥ÅJÿ bE "Purity/0FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPÇúì»J€ bE "Purity/1FRt*SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPe]ÍJ bE "Purity/4FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP÷¬¿J‚ bE "Purity/*FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP ÌW€Jƒ bE "Purity/*FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP“èŸ3H„ bC Purity/*FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JPCÞåàH… bC Purity/0FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JPåá_J† bE "Purity/2FRt3SRqqt1FBt/event_purity"trainer/global_step2:JP]šWŒJ‡ bE "Purity/*FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPЕ2RHˆ bC Purity/4FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP‹FzJ‰ bE "Purity/4FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP8ädzHŠ bC Purity/1FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPݸí H‹ bC Purity/*FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP&9þNŒ bI &Purity/0FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPÀǬ‡N bI &Purity/1FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP£=€JŽ bE "Purity/1FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPŒã…œH bC Purity/0FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPX¸TòH bC Purity/3FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP;ŃJ‘ bE "Purity/*FRt4SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPy÷J’ bE "Purity/1FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP™´±ªJ“ bE "Purity/1FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP¸/ý J” bE "Purity/3FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP±êH• bC Purity/*FRt4SRqqt1FBt/FBt_purity"trainer/global_step2:JP¬º…H– bC Purity/*FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JPt¾TyJ— bE "Purity/0FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPlKCH˜ bC Purity/*FRt*SRqqt4FBt/FRt_purity"trainer/global_step2:JPšu*J™ bE "Purity/*FRt1SRqqt4FBt/event_purity"trainer/global_step2:JPu2-¯Jš bE "Purity/*FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPªã³OJ› bE "Purity/0FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPúaœNœ bI &Purity/0FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPí{O•H bC Purity/2FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPþF¯ZJž bE "Purity/0FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPïÏŠ°HŸ bC Purity/3FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPô@jH  bC Purity/3FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPÜ/÷—J¡ bE "Purity/1FRt3SRqqt2FBt/event_purity"trainer/global_step2:JPI«IåN¢ bI &Purity/1FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPNY§N£ bI &Purity/2FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP@¤ b; particle/accuracy_4_of_4"trainer/global_step2:JPŒ¿ˆH¥ bC Purity/*FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JP´•°üJ¦ bE "Purity/4FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPùDt N§ bI &Purity/*FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPµ#@öJ¨ bE "Purity/1FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPû¾ÓJ© bE "Purity/1FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP¸8˜Nª bI &Purity/3FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP̱ˆÁN« bI &Purity/4FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPw’ÞÝH¬ bC Purity/4FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPð'zÎH­ bC Purity/0FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPÈ¡Œ6H® bC Purity/1FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPvá‹®J¯ bE "Purity/1FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPñûª#H° bC Purity/1FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP¦6J± bE "Purity/2FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPzw!ÅN² bI &Purity/3FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPÔÝN³ bI &Purity/3FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP ¿e@´ b; particle/accuracy_1_of_7"trainer/global_step2:JPœ¶ ·Jµ bE "Purity/*FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPŸ:½0N¶ bI &Purity/1FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPã׊=N· bI &Purity/4FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPG&N¸ bI &Purity/4FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP”ªo7H¹ bC Purity/2FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JPîv¸\Hº bC Purity/1FRt1SRqqt3FBt/FRt_purity"trainer/global_step2:JPüªùzH» bC Purity/4FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP)8XÓH¼ bC Purity/0FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPÆf?J½ bE "Purity/3FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPí‹ýæH¾ bC Purity/0FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPš¸H¿ bC Purity/1FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPeçŒNÀ bI &Purity/*FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP ü:½JÁ bE "Purity/3FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP›à÷¥J bE "Purity/3FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPÝÊÁHà bC Purity/0FRt4SRqqt1FBt/FBt_purity"trainer/global_step2:JPzàýâJÄ bE "Purity/0FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP_MÇZJÅ bE "Purity/2FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPÌ|IHÆ bC Purity/3FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP³¬©ÛHÇ bC Purity/*FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JP­”Z£JÈ bE "Purity/3FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPRAöJÉ bE "Purity/*FRt2SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPWç1¤HÊ bC Purity/4FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPÊxÍJË bE "Purity/1FRt0SRqqt3FBt/event_purity"trainer/global_step2:JPóæyºNÌ bI &Purity/2FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPî‡àHÍ bC Purity/*FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPpÐaðNÎ bI &Purity/4FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPš*„HÏ bC Purity/0FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JP¹pJHÐ bC Purity/*FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPÄF‘òNÑ bI &Purity/*FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP©I‰HÒ bC Purity/*FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP¾NÓ bI &Purity/4FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP‡úÓÓJÔ bE "Purity/3FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPîu³¿JÕ bE "Purity/*FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP°¼GJÖ bE "Purity/*FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPüØœN× bI &Purity/1FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP\ºãmNØ bI &Purity/1FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPèˆCŽHÙ bC Purity/2FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP»!)|NÚ bI &Purity/2FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPê<†«HÛ bC Purity/3FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPR_~‰JÜ bE "Purity/*FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPkD~NÝ bI &Purity/*FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP†Ê#NÞ bI &Purity/0FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPmœ‚’Hß bC Purity/1FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JPÊè¤Jà bE "Purity/1FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPÅð}ÐJá bE "Purity/1FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPóÛjHâ bC Purity/1FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP¼¿Hã bC Purity/2FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPçÄÚ!ä bE "Purity/2FRt1SRqqt1FBt/evenÎÏÉû)t_purity"trainer/global_step2:JP‡ðY$Nå bI &Purity/*FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP`Û=@Hæ bC Purity/*FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP¾ÛJç bE "Purity/*FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP]~­ÿN¸ bI &Purity/1FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP;z­‰J¹ bE "Purity/2FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP´ãHHº bC Purity/1FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JP¼i¡³N» bI &Purity/0FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPëMH¼ bC Purity/2FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPQêA…J½ bE "Purity/*FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPÃç¿?H¾ bC Purity/0FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JP¢ ¸kJ¿ bE "Purity/1FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP©Çp,NÀ bI &Purity/1FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP¸ÉRNÁ bI &Purity/*FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP^·þN bI &Purity/3FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPs·6ÓJà bE "Purity/*FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPÞ#³HÄ bC Purity/1FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP0é£NÅ bI &Purity/3FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPaX @Æ b; particle/accuracy_1_of_5"trainer/global_step2:JPjÜHÇ bC Purity/*FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JP®9k;È b6 jet/accuracy_5_of_6"trainer/global_step2:JPNܽNÉ bI &Purity/*FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPë,JÊ bE "Purity/1FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPMb\JË bE "Purity/2FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPPÁ‘HÌ bC Purity/1FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JPTH"h;Í b6 jet/accuracy_7_of_7"trainer/global_step2:JPŽ!JÎ bE "Purity/4FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÄÊZ€JÏ bE "Purity/*FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPKcõHÐ bC Purity/0FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP@^NÑ bI &Purity/2FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP±ÜNÒ bI &Purity/2FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPIht;Ó b6 jet/accuracy_1_of_5"trainer/global_step2:JPØ¡K@Ô b; particle/accuracy_2_of_2"trainer/global_step2:JPîõ+`JÕ bE "Purity/0FRt4SRqqt1FBt/event_purity"trainer/global_step2:JPé ÕNÖ bI &Purity/*FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPQúÄH× bC Purity/0FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPFmt:JØ bE "Purity/2FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP|Ç]JÙ bE "Purity/2FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP5“£`NÚ bI &Purity/2FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP?ÊÄJÛ bE "Purity/2FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPå,ã®HÜ bC Purity/4FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPéaŒNÝ bI &Purity/*FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPÕIj&NÞ bI &Purity/*FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPð$òJß bE "Purity/0FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPŒÒNà bI &Purity/1FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPŒÐ$'Há bC Purity/2FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPREQ:;â b6 jet/accuracy_3_of_5"trainer/global_step2:JP3”"Jã bE "Purity/*FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÕvßiHä bC Purity/3FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP“ë«¢Hå bC Purity/*FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPÂY>Næ bI &Purity/3FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPr>ùJç bE "Purity/3FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPŽ-Jè bE "Purity/4FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPR‡šJé bE "Purity/4FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP,.|’Nê bI &Purity/*FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP‘\eHë bC Purity/*FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPK̾,Hì bC Purity/*FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP/Hí bC Purity/1FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPì5}Nî bI &Purity/1FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPüFë©Nï bI &Purity/1FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPÉQ(Jð bE "Purity/0FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPË©±Jñ bE "Purity/*FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPÊt¿oHò bC Purity/2FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP±¥AÿHó bC Purity/3FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPi\9ô b4 particle/accuracy"trainer/global_step2:JPˆ2ŸíJõ bE "Purity/4FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP Ê Nö bI &Purity/0FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPfAˆ’J÷ bE "Purity/2FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPi7¨iHø bC Purity/*FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JP#\lúNù bI &Purity/0FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPQšàNú bI &Purity/3FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP-‡Hû bC Purity/2FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPMVòˆHü bC Purity/*FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP™7­‡Hý bC Purity/1FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPž-·Nþ bI &Purity/2FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP 5C6Hÿ bC Purity/2FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPeDzH€ bC Purity/4FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP» }©J bE "Purity/0FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP-p~J‚ bE "Purity/0FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPÓô@ƒ b; particle/accuracy_4_of_5"trainer/global_step2:JP².Â;„ b6 jet/accuracy_3_of_6"trainer/global_step2:JPú¦H… bC Purity/3FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPqä;\H† bC Purity/1FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPšƒîXH‡ bC Purity/2FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPƺÆWNˆ bI &Purity/2FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPR¸‡J‰ bE "Purity/3FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP—E£NŠ bI &Purity/4FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPÐ ´;‹ b6 jet/accuracy_1_of_1"trainer/global_step2:JPGз<Œ b7 particle/specificity"trainer/global_step2:JPŠ–EÑN bI &Purity/*FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPä"NŽ bI &Purity/0FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPÝÚ]KN bI &Purity/1FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP|™pN bI &Purity/1FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPåœPN‘ bI &Purity/1FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP`%H’ bC Purity/1FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPz ºJ“ bE "Purity/2FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP‘ç‚N” bI &Purity/3FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP¼—Ž¿J• bE "Purity/*FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPv|HN– bI &Purity/0FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP;e;— b6 jet/accuracy_1_of_2"trainer/global_step2:JP:ì—H˜ bC Purity/3FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP¨¡’J™ bE "Purity/2FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP!=ÊHš bC Purity/4FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP@{ìN› bI &Purity/0FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPµ§¸yNœ bI &Purity/1FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP&3Ì$H bC Purity/0FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JPNˆÈNž bI &Purity/3FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPçMöNŸ bI &Purity/4FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP>Í;  b6 jet/accuracy_5_of_5"trainer/global_step2:JP%àRòH¡ bC Purity/*FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JP~ÝÏJ¢ bE "Purity/0FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP=Re‹J£ bE "Purity/0FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP<Í`J¤ bE "Purity/1FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPäóú(;¥ b6 jet/accuracy_2_of_6"trainer/global_step2:JPv䈜H¦ bC Purity/*FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPŸ¾ kJ§ bE "Purity/1FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPÁßÙN¨ bI &Purity/4FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP<å @© b; particle/accuracy_1_of_4"trainer/global_step2:JPO„‰Jª bE "Purity/3FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÓk0&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9889500737190247 6 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.268410861492157 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.12109699100255966 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.26568084955215454 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9933110475540161 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2143678218126297 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.3650672435760498 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.21141976118087769 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9767658710479736 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.35135984420776367 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.16750842332839966 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,particle/sensitivity‚0.21388761699199677 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13120567798614502 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.31246888637542725 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.20038312673568726 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8399626612663269 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.13005363941192627 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.12109699100255966 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900095462799072 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9955357313156128 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.27649208903312683 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008629263378679752 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.2872483432292938 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6787656545639038 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.23927415907382965 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15137773752212524 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.3859127163887024 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 particle/accuracy_4_of_6‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.47158128023147583 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.2291666865348816 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.5454545617103577 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.19778281450271606 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.07173644751310349 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.1576576679944992 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 particle/accuracy_3_of_5‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.12773756682872772 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006554915104061365 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.28420567512512207 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FRt_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.2730768024921417 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9898083209991455 ( Purity/2FRt4SRqqt0FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9783831834793091 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.227373868227005 particle/accuracy_6_of_6‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.29530954360961914 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.275369256734848 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9285714030265808 *jet/accuracy_1_of_3‚0.7766847610473633 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.17105263471603394 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2202533781528473 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285715222358704 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.171875 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.21089325845241547 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.2382994294166565 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.991608738899231 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 , Purity/3FRt0SRqqt1FBt/FRt_purity‚0.28125 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4464285671710968 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.19743452966213226 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1944376677274704 :"Purity/1FRt*SRqqt3FBt/event_purity‚0.02777777798473835 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.01677374541759491 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 _runtime‚1475.515896875 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.20033235847949982 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.33546462655067444 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.41999998688697815 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9853864312171936 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.25832802057266235 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.49433013796806335 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3032838702201843 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9891295433044434 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.10421497374773026 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 particle/accuracy_5_of_5‚1 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 6 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.243763729929924 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.19386766850948334 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.33666956424713135 +jet/accuracy_2_of_2‚0.08758573979139328 :"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.48245614767074585 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.44700708985328674 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.306644082069397 :"Purity/2FRt0SRqqt1FBt/event_puritƒQ ­ay‚0.04449405148625374 6 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.262037068605423 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.16936145722866058 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2291518896818161 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.016115175560116768 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 jet/accuracy_1_of_6‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9946319460868835 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9704861044883728 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9912281036376953 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.2836950123310089 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1340898871421814 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.38678163290023804 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1555555760860443 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.25651270151138306 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08121684193611145 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3949897289276123 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1979801207780838 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.2698853313922882 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.2850489318370819 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9078947305679321 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07394369691610336 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5871795415878296 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9599999785423279 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1785714328289032 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9883720874786377 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.1875 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2083333432674408 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 particle/accuracy_4_of_4‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5875000357627869 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9961822032928467 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3087451756000519 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.2795437276363373 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9809523820877075 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.31334003806114197 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0031746032182127237 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9879387021064758 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 % _timestamp‚1.7884654223287468e+09 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9655171632766724 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.2606718838214874 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.31896552443504333 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8050816655158997 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.03125 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.3055555522441864 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 - Purity/0FRt0SRqqt3FBt/FBt_purity‚0.921875 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.22128097712993622 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2369213104248047 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.3522322475910187 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.1943160444498062 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21533094346523285 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.18329426646232605 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.3125 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.013437933288514614 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03077767603099346 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2539921998977661 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.27760419249534607 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9857854843139648 8"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.557238757610321 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.32170942425727844 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.28576236963272095 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.05464988574385643 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.41273584961891174 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 particle/accuracy_7_of_7‚0.5 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 _step‚98 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 +"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.25 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.022042015567421913 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.22023391723632812 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.2967836558818817 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.1892787516117096 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 'particle/f_score‚0.2885393500328064 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.14814814925193787 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3866666853427887 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.21579520404338837 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 jet/accuracy_6_of_7‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4004574716091156 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.24111758172512054 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 particle/accuracy_3_of_4‚1 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.16203702986240387 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 epoch‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.320231169462204 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 jet/accuracy_5_of_7‚0 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.28947368264198303 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.28813913464546204 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9991645812988281 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.01709555834531784 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3382440507411957 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2291666716337204 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.40777134895324707 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.19617116451263428 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.15223446488380432 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.11353456228971481 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.3055555522441864 jet/accuracy_5_of_6‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.2561038136482239 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.4640582203865051 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 jet/accuracy_7_of_7‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 trainer/global_step‚1208 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.006009316071867943 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9802913069725037 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.8741230964660645 particle/accuracy_2_of_2‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9818881750106812 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.2775525152683258 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.35321658849716187 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 6 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.995634913444519 +jet/accuracy_3_of_5‚0.06804275512695312 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.15432099997997284 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9929137825965881 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.2916666567325592 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.23674407601356506 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.23236775398254395 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.44590815901756287 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.07407407462596893 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.35672518610954285 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18397356569766998 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.14435461163520813 (particle/accuracy‚0.8354104161262512 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9677227139472961 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.27855417132377625 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.009433962404727936 particle/accuracy_4_of_5‚1 jet/accuracy_3_of_6‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2083333432674408 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.29201388359069824 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.27960526943206787 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.3869451880455017 *particle/specificity‚0.950417697429657 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.2592731714248657 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.019274376332759857 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.15946654975414276 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 *jet/accuracy_1_of_2‚0.6564766764640808 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.01313900388777256 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.2475375533103943 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.4423076808452606 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3565866947174072 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.0733773410320282 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_4‚1 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 particle/accuracy_3_of_7‚1 particle/accuracy_4_of_7‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2053571492433548 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.997683584690094 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.30222073197364807 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5085169672966003 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5392857193946838 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.21994948387145996 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.007241893094033003 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3032838702201843 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.2777777910232544 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.3473441004753113 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 particle/accuracy_5_of_7‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.31597065925598145 particle/accuracy_2_of_3‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.3819444477558136 jet/accuracy_4_of_6‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.008963856846094131 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_5‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.5303030610084534 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.2125091254711151 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 -jet/accuracy_4_of_4‚0.0019400353776291013 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.07575757801532745 particle/accuracy_6_of_7‚1 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03814643993973732 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.3235294222831726 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.25517240166664124 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.27272728085517883 particle/accuracy_5_of_6‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.18007661402225494 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40356504917144775 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.276635080575943 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.28996360301971436 *jet/accuracy_3_of_4‚0.0593998059630394 jet/accuracy_4_of_5‚0 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18893758952617645 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3794483542442322 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.24046482145786285 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 particle/accuracy_2_of_4‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9882201552391052 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9841780662536621 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1349206417798996 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_2_of_6‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4537646770477295 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.49326685070991516 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.224893257021904 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.3114859163761139 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1111111119389534 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 8"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.546172559261322 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_3‚1 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.1875 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.45985502004623413 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.31790125370025635 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_7‚1 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.19629204273223877 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.11666665971279144 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.982020914554596 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.2113126665353775 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2291666716337204 jet/accuracy_4_of_7‚0¢Ïàãá Ť 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.2872483432292938 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6787656545639038 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.23927415907382965 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15137773752212524 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.3859127163887024 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 particle/accuracy_4_of_6‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.47158128023147583 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.2291666865348816 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.5454545617103577 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.19778281450271606 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.07173644751310349 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.1576576679944992 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 particle/accuracy_3_of_5‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.12773756682872772 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006554915104061365 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.28420567512512207 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FRt_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.2730768024921417 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9898083209991455 ( Purity/2FRt4SRqqt0FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9783831834793091 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.227373868227005 particle/accuracy_6_of_6‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.29530954360961914 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.275369256734848 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9285714030265808 *jet/accuracy_1_of_3‚0.7766847610473633 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.17105263471603394 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2202533781528473 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285715222358704 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.171875 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.21089325845241547 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.2382994294166565 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.991608738899231 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 , Purity/3FRt0SRqqt1FBt/FRt_purity‚0.28125 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4464285671710968 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.19743452966213226 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1944376677274704 :"Purity/1FRt*SRqqt3FBt/event_purity‚0.02777777798473835 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.01677374541759491 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 _runtime‚1475.515896875 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.20033235847949982 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.33546462655067444 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.41999998688697815 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9853864312171936 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.25832802057266235 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.49433013796806335 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3032838702201843 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9891295433044434 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.10421497374773026 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 particle/accuracy_5_of_5‚1 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 6 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.243763729929924 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.19386766850948334 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.33666956424713135 +jet/accuracy_2_of_2‚0.08758573979139328 :"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.48245614767074585 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.44700708985328674 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.306644082069397 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.04449405148625374 6 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.262037068605423 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.16936145722866058 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2291518896818161 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.016115175560116768 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 jet/accuracy_1_of_6‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9946319460868835 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9704861044883728 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9912281036376953 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 .œ„uis@&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.2836950123310089 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1340898871421814 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.38678163290023804 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1555555760860443 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.25651270151138306 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08121684193611145 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3949897289276123 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1979801207780838 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.2698853313922882 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.2850489318370819 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9078947305679321 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07394369691610336 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5871795415878296 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9599999785423279 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1785714328289032 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9883720874786377 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.1875 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2083333432674408 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 particle/accuracy_4_of_4‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5875000357627869 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9961822032928467 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3087451756000519 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.2795437276363373 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9809523820877075 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.31334003806114197 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0031746032182127237 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9879387021064758 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 % _timestamp‚1.7884654223287468e+09 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9655171632766724 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.2606718838214874 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.31896552443504333 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8050816655158997 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.03125 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.3055555522441864 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 - Purity/0FRt0SRqqt3FBt/FBt_purity‚0.921875 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.22128097712993622 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2369213104248047 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.3522322475910187 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.1943160444498062 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21533094346523285 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.18329426646232605 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.3125 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.013437933288514614 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03077767603099346 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2539921998977661 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.27760419249534607 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9857854843139648 8"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.557238757610321 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.32170942425727844 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.28576236963272095 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.05464988574385643 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.41273584961891174 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 particle/accuracy_7_of_7‚0.5 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 _step‚98 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 +"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.25 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.022042015567421913 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.22023391723632812 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.2967836558818817 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.1892787516117096 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 'particle/f_score‚0.2885393500328064 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.14814814925193787 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3866666853427887 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.21579520404338837 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 jet/accuracy_6_of_7‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4004574716091156 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.24111758172512054 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 particle/accuracy_3_of_4‚1 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.16203702986240387 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 epoch‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.320231169462204 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 jet/accuracy_5_of_7‚0 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.28947368264198303 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.28813913464546204 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9991645812988281 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.01709555834531784 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3382440507411957 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2291666716337204 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.40777134895324707 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.19617116451263428 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.15223446488380432 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.11353456228971481 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.3055555522441864 jet/accuracy_5_of_6‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.2561038136482239 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.4640582203865051 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 jet/accuracy_7_of_7‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 trainer/global_step‚1208 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.006009316071867943 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9802913069725037 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.8741230964660645 particle/accuracy_2_of_2‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9818881750106812 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.2775525152683258 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.35321658849716187 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 6 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.995634913444519 +jet/accuracy_3_of_5‚0.06804275512695312 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.15432099997997284 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9929137825965881 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.2916666567325592 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.23674407601356506 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.23236775398254395 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.44590815901756287 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.07407407462596893 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.35672518610954285 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18397356569766998 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.14435461163520813 (particle/accuracy‚0.8354104161262512 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9677227139472961 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.27855417132377625 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.009433962404727936 particle/accuracy_4_of_5‚1 jet/accuracy_3_of_6‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2083333432674408 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.29201388359069824 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.27960526943206787 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.3869451880455017 *particle/specificity‚0.950417697429657 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.2592731714248657 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.019274376332759857 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.15946654975414276 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 *jet/accuracy_1_of_2‚0.6564766764640808 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.01313900388777256 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.2475375533103943 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.4423076808452606 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3565866947174072 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.0733773410320282 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_4‚1 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 particle/accuracy_3_of_7‚1 particle/accuracy_4_of_7‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2053571492433548 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.997683584690094 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.30222073197364807 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5085169672966003 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5392857193946838 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.21994948387145996 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.007241893094033003 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3032838702201843 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.2777777910232544 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.3473441004753113 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 particle/accuracy_5_of_7‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.31597065925598145 particle/accuracy_2_of_3‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.3819444477558136 jet/accuracy_4_of_6‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.008963856846094131 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_5‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.5303030610084534 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.2125091254711151 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 -jet/accuracy_4_of_4‚0.0019400353776291013 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.07575757801532745 particle/accuracy_6_of_7‚1 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03814643993973732 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.3235294222831726 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.25517240166664124 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.27272728085517883 particle/accuracy_5_of_6‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.18007661402225494 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40356504917144775 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.276635080575943 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.28996360301971436 *jet/accuracy_3_of_4‚0.0593998059630394 jet/accuracy_4_of_5‚0 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18893758952617645 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3794483542442322 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.24046482145786285 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 particle/accuracy_2_of_4‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9882201552391052 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9841780662536621 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1349206417798996 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_2_of_6‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4537646770477295 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.49326685070991516 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.224893257021904 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.3114859163761139 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1111111119389534 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 8"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.546172559261322 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_3‚1 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.1875 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.45985502004623413 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.31790125370025635 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_7‚1 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.19629204273223877 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.11666665971279144 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.982020914554596 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.2113126665353775 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2291666716337204 jet/accuracy_4_of_7‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6278747320175171 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ( Purity/2FRt4SRqqt*FBt/FRt_purity‚0.5 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 6 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.988304078578949 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19390489161014557 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_7‚1 jet/accuracy_2_of_7‚0.5 +"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.25 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9219858646392822 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.19794048368930817 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 particle/accuracy_3_of_3‚1 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.24628068506717682 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5081075429916382 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 jet/accuracy_3_of_7‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.2916666567325592 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11123470216989517 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 particle/accuracy_1_of_2‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9960317611694336 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_6_of_6‚0 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 )loss/val_total_loss‚54.11787007178988 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1376735270023346 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9889500737190247 6 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.268410861492157 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.12109699100255966 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.26568084955215454 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9933110475540161 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2143678218126297 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.3650672435760498 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.21141976118087769 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9767658710479736 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.35135984420776367 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.16750842332839966 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,particle/sensitivity‚0.21388761699199677 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13120567798614502 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.31246888637542725 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.20038312673568726 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8399626612663269 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.13005363941192627 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.12109699100255966 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900095462799072 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9955357313156128 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.27649208903312683 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008629263378679752 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625b3jJ=‘â ‚(:† —¢çÔ›øȈ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚122.486' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.6215 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Còpã ‚(:å —¢çÔ‡ý…‰ disk.md0p1.out‚ 154.6484375 disk.rbd2.out‚ 0.279296875 disk.sda.out‚160.01318359375' proc.memory.availableMB‚ 9892.140625 disk.rbd0.out‚ 0.279296875( disk./.usagePercent‚83.1673288721357 disk.sdb.out‚160.01318359375! disk.sdb2.out‚160.01318359375$ disk./.usageGB‚2974.466522216797 disk.nvme1n1.in‚ 260.14453125# disk.nvme1n1.out‚10380.15234375" disk.nvme0n1.out‚ 8694.92578125 disk.rbd2.in‚0 disk.dm-0.in‚ 590.86328125 disk.sda2.in‚ 0.27734375! disk.sda2.out‚160.01318359375 disk.sdb2.in‚ 0.15234375 disk.md0.out‚ 154.6484375 disk.rbd0.in‚0.015625% memory_percent‚39.623165130615234# proc.memory.rssMB‚ 3044.58984375 cpu‚25.04636195464663 network.sent‚ 1.810918e+06 disk.sdb.in‚ 0.15234375 disk.sdb1.out‚0 disk.dm-0.out‚ 19075.078125 disk.md0.in‚ 0.4296875 disk.sda.in‚ 0.27734375 disk.nvme0n1.in‚ 330.71875 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚18.582701683044434 proc.cpu.threads‚17 network.recv‚540454 disk.sdb1.in‚0 disk.md0p1.in‚ 0.4296875û«ºnä ‚(:ã ¦¢çÔï–‡ disk.nvme0n1.in‚ 330.71875 disk.sda1.in‚0 disk.md0p1.out‚ 156.5390625 disk.md0.in‚ 0.4296875 proc.memory.rssMB‚ 3044.59375 disk.dm-0.in‚ 590.86328125 disk.dm-0.out‚ 19475.90625* proc.memory.percent‚18.582725524902344) disk./.usagePercent‚83.16591306023231$ disk./.usageGB‚2974.415885925293 disk.sda1.out‚0 disk.md0.out‚ 156.5390625 disk.rbd2.in‚0 disk.nvme1n1.in‚ 260.14453125 disk.sdb.in‚ 0.15234375 disk.sdb1.in‚0 network.sent‚ 1.923171e+06! disk.sda2.out‚161.96630859375 disk.rbd2.out‚ 0.279296875 disk.sda.out‚161.96630859375) proc.memory.availableMB‚ 9892.18359375 network.recv‚545902 disk.sda2.in‚ 0.27734375 disk.md0p1.in‚ 0.4296875 disk.sdb2.in‚ 0.15234375 disk.sdb.out‚161.96630859375 disk.sdb1.out‚0 disk.sda.in‚ 0.27734375$ memory_percent‚39.62290287017822 disk.rbd0.in‚0.015625" disk.nvme0n1.out‚ 8695.79296875! disk.sdb2.out‚161.96630859375 disk.rbd0.out‚ 0.279296875 cpu‚25.04629843527955 proc.cpu.threads‚17# disk.nvme1n1.out‚10780.11328125¡j&@å ‚(:… ¦¢çԎ㉈ gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚139.868' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.967 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e$ØÈnæ ‚(:á µ¢çÔ½½ý† network.sent‚ 1.926709e+06 disk.nvme0n1.in‚ 330.71875 disk.dm-0.out‚ 196R7¸þ89.6875 proc.cpu.threads‚17$ disk./.usageGB‚2974.497097015381 disk.sdb.out‚163.6572265625 disk.sdb1.in‚0 disk.sdb2.out‚163.6572265625 disk.rbd2.in‚0 cpu‚25.04585659661667 disk.md0p1.out‚ 158.18359375 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 0.279296875 disk.nvme1n1.in‚ 260.14453125" disk.nvme1n1.out‚ 10993.8359375) disk./.usagePercent‚83.16818375630065 disk.md0.in‚ 0.4296875 disk.md0.out‚ 158.18359375) proc.memory.availableMB‚ 9891.60546875 network.recv‚549811 disk.sda.out‚163.6572265625 disk.sdb.in‚ 0.15234375 disk.rbd2.out‚ 0.279296875" proc.memory.rssMB‚ 3044.6015625 disk.sdb1.out‚0 disk.dm-0.in‚ 590.86328125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚163.6572265625% memory_percent‚39.626431465148926* proc.memory.percent‚18.582773208618164 disk.sda.in‚ 0.27734375! disk.nvme0n1.out‚ 8695.8515625 disk.sda2.in‚ 0.27734375 disk.md0p1.in‚ 0.4296875 disk.sdb2.in‚ 0.15234375úèç ‚(:… µ¢çÔ‰¤™ˆ gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚112.572' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.143 gpu.0.smClock‚1245 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ s2è - _wandbruntime‚1525  _runtime‚1525é¿-›Æé À % _timestamp‚1.7884654725903912e+09 _runtime‚1525.775476618 _step‚99 & lr-AdamW/pg1‚0.0008244252116187588 & lr-AdamW/pg2‚0.0008244252116187588 trainer/global_step‚1249Ò]TÊê Ä & lr-AdamW/pg1‚0.0008244252116187588 & lr-AdamW/pg2‚0.0008244252116187588 trainer/global_step‚1249 % _timestamp‚1.7884654725903912e+09 _runtime‚1525.775476618 _step‚99cŠ|êR2ë - _wandbruntime‚1525  _runtime‚1525Nê SÆ'ì ÀO BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚53503607997952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚53503607997952 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚48536793648656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚51019417523200 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚689 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57851413 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚48536810688016 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23825307 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚53503859668480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34026106 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚54034852497920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚53503859668480 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2484190474752 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚54034876438016 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14247186 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9046537 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33999764 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚56564615857664 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23825305 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚51019417523200 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325046272 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2529763359744 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0à™e‚Ê'í ÄO % _timestamp‚1.7884654725912738e+09 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚53503607997952 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚48536793648656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚51019417523200 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚689 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57851413 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚48536810688016 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23825307 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚53503859668480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34026106 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚54034852497920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚53503859668480 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2484190474752 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚54034876438016 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14247186 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9046537 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33999764 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚56564615857664 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23825305 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚51019417523200 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325046272 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2529763359744 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚53503607997952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290607104 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34052824 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2486154500608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚51058721563136 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚654 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚51053429121199 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57896406 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314547200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2486154500608 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚656 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57870036 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23843582 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23843582 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _step‚101 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23311694 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚56608295181824 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2532049320960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23843580 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2531758713856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚51058721563136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9053708 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2479334096403 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚54076536467968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23843580 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚48574341045440 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚53545109744640”·Ö! ð ÍM WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57896406 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚48574324006080 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H›Sëæ3%@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290607104 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34052824 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2486154500608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚51058721563136 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚654 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚51053429121199 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57896406 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314547200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2486154500608 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚656 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57870036 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23843582 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23843582 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _step‚101 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23311694 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚56608295181824 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2532049320960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23843580 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2531758713856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚51058721563136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9053708 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2479334096403 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚54076536467968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23843580 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚48574341045440 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚53545109744640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23312350 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2486388181504eþ‰Ó˜2ñ - _wandbruntime‚1526  _runtime‚1526z>loTò Î trainer/global_step‚1249 % _timestamp‚1.7884654736768456e+09 /loss/FRt2/assignment_loss‚4.704699662753514 0loss/FRt3/assignment_loss‚1.8621337026880498 2loss/SRqqt2/assignment_loss‚1.9500774416056545 /loss/FRt3/detection_loss‚1.8189861805815446 .loss/FRt4/detection_loss‚2.885399749202113 epoch‚1 /loss/FRt1/assignment_loss‚3.871963771906766 2loss/SRqqt1/assignment_loss‚1.6424609328950606 1loss/SRqqt1/detection_loss‚1.7186685233346877 .loss/FBt1/detection_loss‚2.396062910556793 0loss/FBt2/assignment_loss‚0.7504052817821503 .loss/FRt1/detection_loss‚2.185749405622482 .loss/FRt2/detection_loss‚2.451185200895582 1loss/SRqqt3/detection_loss‚0.9327402924902217 /loss/FRt4/assignment_loss‚5.213165160148375 .loss/FBt2/detection_loss‚3.633639335632324 %loss/total_loss‚52.71257795049861 0loss/FBt4/assignment_loss‚0.8097196268538635 0loss/SRqqt4/detection_loss‚2.858125627040863 /loss/FBt3/detection_loss‚0.8348440833968198 /loss/FBt4/detection_loss‚2.3388574282328287 _runtime‚1526.861597206 _step‚102 1loss/SRqqt3/assignment_loss‚1.710425573206225 1loss/SRqqt2/detection_loss‚2.6398969437136794 1loss/SRqqt4/assignment_loss‚2.641193801537156 0loss/FBt1/assignment_loss‚0.5263817872182699 0loss/FBt3/assignment_loss‚0.3357955272035952J®òXó Ò 1loss/SRqqt2/detection_loss‚2.6398969437136794 1loss/SRqqt4/assignment_loss‚2.641193801537156 0loss/FBt1/assignment_loss‚0.5263817872182699 0loss/FBt3/assignment_loss‚0.3357955272035952 trainer/global_step‚1249 % _timestamp‚1.7884654736768456e+09 /loss/FRt2/assignment_loss‚4.704699662753514 0loss/FRt3/assignment_loss‚1.8621337026880498 2loss/SRqqt2/assignment_loss‚1.9500774416056545 /loss/FRt3/detection_loss‚1.8189861805815446 .loss/FRt4/detection_loss‚2.885399749202113 epoch‚1 /loss/FRt1/assignment_loss‚3.871963771906766 2loss/SRqqt1/assignment_loss‚1.6424609328950606 1loss/SRqqt1/detection_loss‚1.7186685233346877 .loss/FBt1/detection_loss‚2.396062910556793 0loss/FBt2/assignment_loss‚0.7504052817821503 .loss/FRt1/detection_loss‚2.185749405622482 .loss/FRt2/detection_loss‚2.451185200895582 1loss/SRqqt3/detection_loss‚0.9327402924902217 /loss/FRt4/assignment_loss‚5.213165160148375 .loss/FBt2/detection_loss‚3.633639335632324 %loss/total_loss‚52.71257795049861 0loss/FBt4/assignment_loss‚0.8097196268538635 0loss/SRqqt4/detection_loss‚2.858125627040863 /loss/FBt3/detection_loss‚0.8348440833968198 /loss/FBt4/detection_loss‚2.3388574282328287 _runtime‚1526.861597206 _step‚102 1loss/SRqqt3/assignment_loss‚1.710425573206225f[`Ž«’ô ‚(:‡ Ä¢çÔœšÃˆ gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚111.955' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚27.98875 gpu.0.smClock‚1260 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úxåßoõ ‚(:ä Ä¢çÔ݈ؖ disk.sda2.out‚165.4482421875 disk.md0p1.in‚ 0.4296875 disk.md0.out‚ 159.92578125$ memory_percent‚39.62728977203369 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.dm-0.out‚19854.87109375 network.recv‚554430) disk./.usagePercent‚83.16012299324609 disk.nvme1n1.in‚ 260.14453125 disk.rbd2.in‚0 disk.sda.out‚165.4482421875 disk.sda2.in‚ 0.27734375 disk.md0.in‚ 0.4296875" proc.memory.rssMB‚ 3044.6015625 disk.sdb.out‚165.4482421875 disk.sda.in‚ 0.27734375 disk.dm-0.in‚ 590.86328125 disk.sdb2.in‚ 0.15234375 disk.sdb2.out‚165.4482421875 disk.rbd2.out‚ 0.279296875) proc.memory.availableMB‚ 9891.46484375* proc.memory.percent‚18.582773208618164% disk./.usageGB‚2974.2088050842285 disk.nvme0n1.in‚ 330.71875 disk.md0p1.out‚ 159.92578125 network.sent‚ 1.945656e+06 disk.rbd0.out‚ 0.279296875 cpu‚25.0455178248866 disk.rbd0.in‚0.015625 disk.sdb.in‚ 0.15234375 disk.sdb1.out‚0" disk.nvme0n1.out‚ 8696.10546875 disk.sda1.in‚0 disk.sda1.out‚0! disk.nvme1n1.out‚ 11158.765625ëu¸Óö ‚(:… Ó¢çÔ½ˆ£† gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚104.826' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚26.2065 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èýq÷ ‚(:æ Ó¢çÔ£›êˆ$ disk./.usageGB‚2974.396682739258" disk.nvme0n1.out‚ 8696.15234375 disk.rbd0.out‚ 0.279296875$ memory_percent‚39.62738513946533" proc.memory.rssMB‚ 3044.6015625 network.recv‚559046 disk.sdb.in‚ 0.15234375 disk.md0.out‚ 161.7578125 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 0.279296875 disk.sda.out‚167.34423828125 disk.sda2.in‚ 0.27734375 disk.md0p1.out‚ 161.7578125 disk.rbd2.in‚0 network.sent‚ 2.041845e+06 disk.sdb.out‚167.34423828125 disk.nvme0n1.in‚ 330.71875 disk.sdb2.in‚ 0.15234375! disk.sdb2.out‚167.34423828125# disk.nvme1n1.out‚11350.44140625 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.dm-0.in‚ 590.86328125 disk.sda1.out‚0 disk.md0p1.in‚ 0.4296875 disk.md0.in‚ 0.4296875 disk.nvme1n1.in‚ 260.14453125) proc.memory.availableMB‚ 9891.44921875 cpu‚25.045557031794466 disk.dm-0.out‚ 20046.59375) disk./.usagePercent‚83.16537613111362 disk.sdb1.in‚0 disk.sda.in‚ 0.27734375! disk.sda2.out‚167.34423828125 disk.sda1.in‚0* proc.memory.percent‚18.582773208618164Ãam:ø ‚(:… â¢çÔåÍЇ gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.228' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚27.057 gpu.0.smClock‚1230 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á±Ëgù ‚(:Ü â¢çÔ¦œ¨ˆ disk.sda2.in‚ 0.27734375 disk.sdb2.in‚ 0.15234375! disk.sdb2.out‚168.88525390625 disk.rbd2.in‚0$ memory_percent‚39.62886333465576) proc.memory.availableMB‚ 9891.20703125 cpu‚25.045302141085298! disk.sda2.out‚168.88525390625 disk.md0p1.in‚ 0.4296875 disk.md0p1.out‚163.25 disk.rbd0.in‚0.015625 disk.dm-0.in‚ 590.86328125) disk./.usagePercent‚83.17238031493697 disk.rbd2.out‚ 0.279296875 disk.sdb.in‚ 0.15234375 disk.sdb1.out‚0 disk.dm-0.out‚ 20297.90625 disk.md0.out‚163.25 disk.sda1.in‚0 disk.nvme1n1.in‚ 260.14453125 disk.sda.in‚ 0.27734375" disk.nvme0n1.out‚ 8696.76953125" proc.memory.rssMB‚ 3044.6015625 disk.sda1.out‚0 disk.nvme0n1.in‚ 330.71875* proc.memory.percent‚18.582773208618164 proc.cpu.threads‚17 network.recv‚563609$ disk./.usageGB‚2974.647186279297 disk.sdb.out‚168.88525390625 disk.sdb1.in‚0 disk.sda.out‚168.88525390625 disk.md0.in‚ 0.4296875 disk.rbd0.out‚ 0.279296875# disk.nvme1n1.out‚11601.13671875 network.sent‚ 2.045986e+06ÐœúUú ‚(:… ñ¢çÔçÍ¡‡ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚110.396' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚27.599 gpu.0.smClock‚1245 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0AèÆÁfû ‚(:Û ñ¢çÔí£¨ˆ disk.rbd2.out‚ 0.279296875 disk.sdb1.out‚0$ disk./.usageGB‚2974.646167755127 disk.md0.in‚ 0.4296875 disk.md0.out‚ 164.4296875( proc.memory.availableMB‚ 9891.4453125* proc.memory.percent‚18.582773208618164 disk.dm-0.in‚ 590.86328125" proc.memory.rssMB‚ 3044.6015625 cpu‚25.045190424260994( disk./.usagePercent‚83.1723518365749 disk.dm-0.out‚20548.25 disk.sda1.in‚0$ memory_percent‚39.62740898132324 disk.sda.in‚ 0.27734375 disk.nvme0n1.out‚ 8696.84375 disk.sdb2.in‚ 0.15234375 disk.rbd0.out‚ 0.279296875 disk.sdb.out‚170.12255859375 disk.sdb.in‚ 0.15234375 network.sent‚ 2.052585e+06 disk.nvme0n1.in‚ 330.71875 disk.sda1.out‚0! disk.sdb2.out‚170.12255859375 disk.nvme1n1.out‚ 11851.40625 proc.cpu.threads‚17 network.recv‚567077! disk.sda2.out‚170.12255859375 disk.md0p1.out‚ 164.4296875 disk.nvme1n1.in‚ 260.14453125 disk.sdb1.in‚0 disk.sda.out‚170.12255859375 disk.md0p1.in‚ 0.4296875 disk.rbd0.in‚0.015625 disk.sda2.in‚ 0.27734375 disk.rbd2.in‚0m¹¦†2ü - _wandbruntime‚1583  _runtime‚1583…qþ÷Åý ¿ _runtime‚1583.987803122 _step‚103 % lr-AdamW/pg1‚0.000824313537644701 % lr-AdamW/pg2‚0.000824313537644701 trainer/global_step‚1299 % _timestamp‚1.7884655308028626e+09þŸÂÉþ à % lr-AdamW/pg1‚0.000824313537644701 % lr-AdamW/pg2‚0.000824313537644701 trainer/global_step‚1299 % _timestamp‚1.7884655308028626e+09 _runtime‚1583.987803122 _step‚103gÎ?Ì2ÿ - _wandbruntime‚1583  _runtime‚1583!½QÆ'€ÀO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60076771 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚50413328559216 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚55566377224704 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚55566377224704 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60103115 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2582406181376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14787284 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚58748909808640 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35337716 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2582406181376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35364058 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2629834868224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2629550539264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24180851 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚699 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2575313647717 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308269056 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚58748601539584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 trainer/global_step‚1299 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚52983719372800 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _runtime‚1583.988944068 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24739057 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24739055 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24181552 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9393567 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚56119074940416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚52988396186497 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884655308039334e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35337716 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚701 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚50413311519856 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚52988642206933 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2582639848960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14787286 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24739055 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚52983719372800 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9394266 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35364058 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55566125554176 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0YX5ðÊ ÄO @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2575313647717 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308269056 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚58748601539584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 trainer/global_step‚1299 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚52983719372800 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _runtime‚1583.988944068 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24739057 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24739055 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24181552 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9393567 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚56119074940416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚52988396186497 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884655308039334e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35337716 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚701 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚50413311519856 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚52988642206933 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2582639848960 \ODeviceStatsMonitor.oü(/n_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14787286 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24739055 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚52983719372800 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9394266 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35364058 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55566125554176 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60076771 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚50413328559216 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚55566377224704 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚55566377224704 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60103115 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2582406181376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14787284 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚58748909808640 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35337716 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2582406181376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35364058 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2629834868224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2629550539264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24180851 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚699 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104hî.<ä2‚- _wandbruntime‚1585  _runtime‚1585]èvþÏ&ƒÉM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24198682 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2584370207232 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35390776 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60148108 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚58792282591232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚55607627300864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9401270 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2631547620864 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2577043155627 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60121738 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 trainer/global_step‚1299 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2584603888128 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24757332 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24757330 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9400596 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2631834033664 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚53028131053551 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24757332 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚53023023412736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚50450841877280 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚56160758910464 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412800 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14798088 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚53023023412736 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚50450858916640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60148108 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚55607627300864 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24757330 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35364408 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚53027885032907 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2577272136911 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 _runtime‚1585.165514257 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60148108 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚58792282591232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚55607627300864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9401270 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2631547620864 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2577043155627 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60121738 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 trainer/global_step‚1299 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2584603888128 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24757332 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24757330 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9400596 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2631834033664 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚53028131053551 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24757332 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚53023023412736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚50450841877280 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚56160758910464 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412800 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14798088 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚53023023412736 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚50450858916640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60148108 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚55607627300864 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24757330 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35364408 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚53027885032907 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2577272136911 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 _runtime‚1585.165514257 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24198682 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2584370207232 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35390776 O¢Š‚(:„ £çÔ—ùч gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚92.404' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚23.101 gpu.0.smClock‚1230 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù<å7e‹‚(:Ú £çÔ’®³ˆ disk.sda1.in‚0 network.sent‚ 2.164959e+06 disk.sdb2.in‚ 0.15234375 disk.md0.out‚ 167.8046875# disk.nvme1n1.out‚12293.44140625 disk.sdb.out‚ 173.599609375 disk.sdb1.in‚0( disk./.usagePercent‚83.1723518365749 disk.sdb2.out‚ 173.599609375 disk.rbd0.in‚0.015625 disk.nvme1n1.in‚ 260.14453125 disk.dm-0.out‚20990.64453125 cpu‚25.04497058903076 disk.sdb.in‚ 0.15234375 disk.sda2.out‚ 173.599609375 disk.md0p1.in‚ 0.4296875 disk.sdb1.out‚0* proc.memory.percent‚18.582773208618164 disk.dm-0.in‚ 590.86328125 disk.rbd0.out‚ 0.279296875 disk.nvme0n1.in‚ 330.71875& proc.memory.availableMB‚ 9891.28125" proc.memory.rssMB‚ 3044.6015625$ disk./.usageGB‚2974.646167755127 disk.md0.in‚ 0.4296875 disk.sda.in‚ 0.27734375 disk.sda.out‚ 173.599609375$ memory_percent‚39.62841033935547 disk.rbd2.in‚0 disk.sda2.in‚ 0.27734375 disk.md0p1.out‚ 167.8046875 proc.cpu.threads‚17 network.recv‚576446 disk.rbd2.out‚ 0.279296875 disk.sda1.out‚0 disk.nvme0n1.out‚ 8697.203125<'C‹÷Œ‚(:† ž£çÔÉ؇ gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚95.529' gpu.0.enforcedPowerLimitWatts‚400.0 WƒVÍšgpu.0.powerPercent‚23.88225 gpu.0.smClock‚1260 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·‹óèq‚(:æ ž£çÔ£¥–ˆ disk.rbd2.out‚ 0.279296875 disk.sdb.in‚ 0.15234375 disk.sdb.out‚175.48291015625 disk.sda.in‚ 0.27734375 disk.sda.out‚175.48291015625 disk.sda2.in‚ 0.27734375 disk.nvme0n1.in‚ 330.71875( disk./.usagePercent‚83.1723518365749! disk.sdb2.out‚175.48291015625 disk.rbd0.out‚ 0.279296875 disk.rbd2.in‚0 disk.sdb1.in‚0 disk.dm-0.out‚ 21241.3984375$ memory_percent‚39.62759971618652* proc.memory.percent‚18.582773208618164" disk.nvme0n1.out‚ 8697.43359375 disk.sda1.out‚0( proc.memory.availableMB‚ 9891.4140625 cpu‚25.044872084414504 network.sent‚ 2.169142e+06 disk.rbd0.in‚0.015625 disk.dm-0.in‚ 590.86328125 disk.sda1.in‚0 disk.md0p1.in‚ 0.4296875" proc.memory.rssMB‚ 3044.6015625 disk.md0.in‚ 0.4296875 disk.md0.out‚ 169.6328125# disk.nvme1n1.out‚12543.96484375 disk.sdb1.out‚0 disk.sdb2.in‚ 0.15234375! disk.sda2.out‚175.48291015625 disk.md0p1.out‚ 169.6328125 proc.cpu.threads‚17 disk.nvme1n1.in‚ 260.14453125 network.recv‚581074$ disk./.usageGB‚2974.646167755127šK&’Ž‚(:‡ ­£çÔû±´‡ gpu.0.gpu‚14.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.357' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.83925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ä c‚(:Ø ­£çÔÁô˜ˆ disk.nvme1n1.in‚ 260.14453125 disk.sdb1.out‚0 disk.sda.in‚ 0.27734375$ memory_percent‚39.62881565093994# proc.memory.rssMB‚ 3044.60546875* proc.memory.percent‚18.582797050476074 cpu‚25.044767842410913) disk./.usagePercent‚83.17234629022722# disk.nvme1n1.out‚12852.86328125 disk.nvme0n1.out‚ 8697.484375 disk.md0p1.in‚ 0.4296875 disk.sdb2.in‚ 0.15234375 network.sent‚ 2.174826e+06 disk.dm-0.out‚21550.34765625 disk.sda1.out‚0 disk.rbd0.out‚ 0.279296875 disk.rbd2.out‚ 0.279296875 disk.sda.out‚ 176.8671875 disk.sda2.in‚ 0.27734375 disk.sda2.out‚ 176.8671875) proc.memory.availableMB‚ 9891.21484375 disk.md0.out‚ 170.9609375 disk.rbd2.in‚0 disk.sdb2.out‚ 176.8671875 disk.sdb.out‚ 176.8671875 disk.sda1.in‚0 disk.md0p1.out‚ 170.9609375 proc.cpu.threads‚17 network.recv‚585295$ disk./.usageGB‚2974.645969390869 disk.rbd0.in‚0.015625 disk.nvme0n1.in‚ 330.71875 disk.dm-0.in‚ 590.86328125 disk.sdb.in‚ 0.15234375 disk.sdb1.in‚0 disk.md0.in‚ 0.4296875Èâf2- _wandbruntime‚1642  _runtime‚1642‘“õÇ‘Á & lr-AdamW/pg1‚0.0008241159853186124 & lr-AdamW/pg2‚0.0008241159853186124 trainer/global_step‚1349 % _timestamp‚1.7884655888499062e+09 _runtime‚1642.034934772 _step‚107 ÿ\óË’Å _step‚107 & lr-AdamW/pg1‚0.0008241159853186124 & lr-AdamW/pg2‚0.0008241159853186124 trainer/global_step‚1349 % _timestamp‚1.7884655888499062e+09 _runtime‚1642.034934772kh¿{,2“- _wandbruntime‚1642  _runtime‚1642ExY«Å'”¿O QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25067427 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25652805 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25652805 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25652807 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2673251688225 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚689 _step‚108 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚57628894780928 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚54948039225344 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57628643110400 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58203273442816 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15327386 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36702010 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚54948021222400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2673022707149 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2680855555584 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚60932941015040 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚58203249502720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚54948021222400 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25652805 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25652807 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2673251688225 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚689 _step‚108 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚57628894780928 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚54948039225344 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57628643110400 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58203273442816 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15327386 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36702010 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚54948021222400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2673022707149 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2680855555584 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚60932941015040 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚58203249502720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273l$-°á2–- _wandbruntime‚1643  _runtime‚1643¬‡yö"—ÊM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25671080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚109 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 trainer/global_step‚1349 _runtime‚1643.214500778 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚60976627193856 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚668 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚55002340944615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58244957412864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58244933472768 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2731669780992 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25084565 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2682585913856 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25671080 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36728728 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚54987325262336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25671082 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15338186 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36728728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2731381271040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚670 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884655900279503e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62399810 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288509952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62399810 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚52327359748480 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62373440 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚60976314743808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚57670144857088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9746379 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2682585913856 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚52327376787840 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2675210177419 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25085235 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62373440 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25671082 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚54987307259392 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚54987307259392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15338188 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚54987325262336 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312450048 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2682819594752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚55002586965259 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_·ÖþÚbatch_end/allocation.small_pool.peak‚32004 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9747047 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2674981196135 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36702360 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62399810 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288509952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62399810 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚52327359748480 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62373440 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚60976314743808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚57670144857088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9746379 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2682585913856 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚52327376787840 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2675210177419 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25085235 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62373440 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25671082 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚54987307259392 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚54987307259392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15338188 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚54987325262336 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312450048 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2682819594752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚55002586965259 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9747047 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2674981196135 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36702360 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 trainer/global_step‚1349 _runtime‚1643.214500778 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚60976627193856 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚668 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚55002340944615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58244957412864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58244933472768 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2731669780992 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25084565 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2682585913856 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25671080 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36728728 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚54987325262336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25671082 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15338186 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36728728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2731381271040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚670 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884655900279503e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚59691412337152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚54166347262256 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38039962 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚56912341074944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26566557 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10087399 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280134656 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2770960747657 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10086734 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚59691412337152 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚59691160666624 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚56912341074944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304074752 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚60287448005120 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2829324635648 trainer/global_step‚1399 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38013620 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15867486 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚56937308009913 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2779071262208 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25954885 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59691160666624 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64606519 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2778837594624 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64580175 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2778837594624 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25954218 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309.c©ÅO XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚56912323072000 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚63116772640768 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2779071262208 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25954885 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 IDeuG׶viceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59691160666624 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64606519 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2778837594624 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64580175 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2778837594624 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25954218 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2779071262208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚56937554030349 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2829604770304 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚54166364301616 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2771189728733 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚59691412337152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚54166347262256 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38039962 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚56912341074944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26566557 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10087399 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280134656 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2770960747657 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10086734 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚59691412337152 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚59691160666624 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚56912341074944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304074752 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚60287448005120 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2829324635648 trainer/global_step‚1399 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38013620 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15867486 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚56937308009913 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25971704 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10093418 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2780801620480 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚63160453889024 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64625142 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚59732662413312 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚60329155915264 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚56951609108992 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2781035301376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26584832 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25972395 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚56951627111936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚60329131975168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚56951609108992 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚59732410729472 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2831321913856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2773148217927 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚56976796856323 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26584830 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352896 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26584830 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2831608326656 _step‚113 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚59732662413312 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38066680 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2772919236643 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2781035301376 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚54203894659040 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38066680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 trainer/global_step‚1399 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26584832 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64651512 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64625142 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15878288 _runtime‚1701.873051853 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10094107 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚63160764241920 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412800 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 trainer/global_step‚1399 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26584832 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64651512 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64625142 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15878288 _runtime‚1701.873051853 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10094107 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚63160764241920 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412800 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25971704 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10093418 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2780801620480 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚63160453889024 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64625142 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚59732662413312 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚60329155915264 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚56951609108992 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2781035301376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26584832 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25972395 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚56951627111936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚60329131975168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚56951609108992 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚59732410729472 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2831321913856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2773148217927 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚56976796856323 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26584830 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352896 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26584830 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2831608326656 _step‚113 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚59732662413312q`À<2­- _wandbruntime‚1701  _runtime‚1701BÙX®Ò /loss/FRt2/assignment_loss‚4.879976757548072 1loss/FBt4/assignment_loss‚0.44613008666781867 /loss/FBt1/detection_loss‚2.8611127614974974 epoch‚1 /loss/FRt3/assignment_loss‚2.137285310142445 0loss/SRqqt1/detection_loss‚1.776207877529992 trainer/global_step‚1399 _step‚114 % _timestamp‚1.7884656486895502e+09 1loss/SRqqt3/detection_loss‚1.0311309270999012 .loss/FBt4/detection_loss‚2.793528604507446 _runtime‚1701.877585043 .loss/FRt3/detection_loss‚2.029784355242297 .loss/FRt4/detection_loss‚3.739568691504629 /loss/FBt2/detection_loss‚2.7616646024915905 /loss/FRt1/assignment_loss‚3.592158936641433 0loss/FBt1/assignment_loss‚0.5750108050415292 2loss/SRqqt2/assignment_loss‚2.0056167443593345 2loss/SRqqt3/assignment_loss‚1.3497047842425458 1loss/FBt2/assignment_loss‚0.41703057578853175 1loss/FBt3/assignment_loss‚0.11095848393682822 /loss/FBt3/detection_loss‚0.8511682819232377 /loss/FRt4/assignment_loss‚4.189274662419369 2loss/SRqqt1/assignment_loss‚1.7300467389443563 2loss/SRqqt4/assignment_loss‚2.1346921768130325 /loss/FRt2/detection_loss‚2.3448990312489597 0loss/SRqqt2/detection_loss‚2.715135721365611 &loss/total_loss‚51.149497659451576 .loss/FRt1/detection_loss‚2.475150153040886 0loss/SRqqt4/detection_loss‚2.202260589454232<=X˜\¯Ö /loss/FRt2/assignment_loss‚4.879976757548072 1loss/FBt4/assignment_loss‚0.44613008666781867 /loss/FBt1/detection_loss‚2.8611127614974974 epoch‚1 /loss/FRt3/assignment_loss‚2.137285310142445 0loss/SRqqt1/detection_loss‚1.776207877529992 trainer/global_step‚1399 _step‚114 % _timestamp‚1.7884656486895502e+09 1loss/SRqqt3/detection_loss‚1.0311309270999012 .loss/FBt4/detection_loss‚2.793528604507446 _runtime‚1701.877585043 .loss/FRt3/detection_loss‚2.029784355242297 .loss/FRt4/detection_loss‚3.739568691504629 /loss/FBt2/detection_loss‚2.7616646024915905 /loss/FRt1/assignment_loss‚3.592158936641433 0loss/FBt1/assignment_loss‚0.5750108050415292 2loss/SRqqt2/assignment_loss‚2.0056167443593345 2loss/SRqqt3/assignment_loss‚1.3497047842425458 1loss/FBt2/assignment_loss‚0.41703057578853175 1loss/FBt3/assignment_loss‚0.11095848393682822 /loss/FBt3/detection_loss‚0.8511682819232377 /loss/FRt4/assignment_loss‚4.189274662419369 2loss/SRqqt1/assignment_loss‚1.7300467389443563 2loss/SRqqt4/assignment_loss‚2.1346921768130325 /loss/FRt2/detection_loss‚2.3448990312489597 0loss/SRqqt2/detection_loss‚2.715135721365611 &loss/total_loss‚51.149497659451576 .loss/FRt1/detection_loss‚2.475150153040886 0loss/SRqqt4/detection_loss‚2.202260589454232rŒXoy°‚(:î ø£çÔ®ûÿˆ disk.sda2.in‚ 0.27734375 disk.md0p1.in‚ 0.4296875 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.sdb1.out‚0 network.sent‚ 2.305436e+06) disk./.usagePercent‚83.15752058292286# proc.memory.rssMB‚ 3044.60546875 disk.sdb.in‚ 0.15234375 disk.sda.in‚ 0.27734375 disk.nvme0n1.in‚ 344.1171875 disk.sda1.in‚0 disk.sdb2.in‚ 0.15234375 disk.md0.in‚ 0.4296875 disk.sdb.out‚184.55126953125 network.recv‚607653 disk.rbd2.in‚0 disk.rbd0.out‚ 0.279296875! disk.sdb2.out‚184.55126953125 cpu‚25.043821483865706 disk.md0.out‚ 178.34765625 disk.rbd2.out‚ 0.279296875 disk.nvme1n1.in‚ 267.66796875# disk.nvme1n1.out‚13841.98828125 disk.dm-0.in‚ 611.78515625 disk.dm-0.out‚ 22540.40625! disk.sda2.out‚184.55126953125 disk.md0p1.out‚ 178.34765625 disk.sda.out‚184.55126953125 disk.rbd0.in‚0.015625* proc.memory.availableMB‚11149.44140625* proc.memory.percent‚18.582797050476074% disk./.usageGB‚2974.1157302856445" disk.nvme0n1.out‚ 8698.41796875% memory_percent‚31.949210166931152 disk.sda1.out‚0\ì’±‚(:‡ ø£çÔß樈 gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚119.701' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.92525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ‘ 9²‚(:‡ ‡¤çÔ‹‹Ò‡ gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚137.969' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.49225 gpu.0.smClock‚1410 gpu.0.mamqþYemoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06¢V=s³‚(:è ‡¤çÔÿñ¿ˆ% disk./.usageGB‚2974.1669006347656 disk.rbd2.in‚0 disk.rbd2.out‚ 0.279296875 disk.sda.in‚ 0.27734375 disk.sda1.out‚0 disk.sdb2.in‚ 0.15234375 disk.rbd0.in‚0.015625 disk.nvme1n1.in‚ 267.671875 disk.sda1.in‚0% memory_percent‚29.367375373840332) disk./.usagePercent‚83.15895132730073 disk.md0.in‚ 0.4296875 disk.sdb.in‚ 0.15234375 disk.sda.out‚186.40478515625* proc.memory.availableMB‚11572.44921875 network.sent‚ 2.414025e+06 disk.md0p1.in‚ 0.4296875" disk.nvme0n1.out‚ 8699.04296875! disk.sda2.out‚186.40478515625# proc.memory.rssMB‚ 3044.60546875 cpu‚25.043462573968654 disk.nvme0n1.in‚346.875 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.dm-0.out‚22674.12890625 network.recv‚612451 disk.md0p1.out‚ 180.14453125 disk.rbd0.out‚ 0.279296875" disk.nvme1n1.out‚ 13975.0859375 disk.sdb.out‚186.40478515625 disk.sdb1.out‚0 disk.sda2.in‚ 0.27734375! disk.sdb2.out‚186.40478515625 disk.md0.out‚ 180.14453125 disk.dm-0.in‚ 614.546875* proc.memory.percent‚18.582797050476074ÄûÎ[´‚(:… –¤çÔÏ㌇ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚139.868' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.967 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00Èerpµ‚(:å –¤çÔ…üˆ disk.rbd2.out‚ 0.279296875 disk.sda1.in‚0 disk.md0p1.in‚5.3125 disk.rbd2.in‚0% memory_percent‚28.478193283081055 disk.sda.in‚ 2.69921875 disk.dm-0.in‚ 755.46484375 disk.sdb.out‚187.88134765625 disk.sda.out‚187.88134765625 disk.rbd0.out‚ 0.279296875 disk.sdb1.out‚0 disk.dm-0.out‚22888.69921875 disk.rbd0.in‚0.015625( disk./.usagePercent‚83.1386310026841" disk.nvme0n1.out‚ 8699.43359375 disk.sda1.out‚0 disk.md0p1.out‚ 181.55078125 disk.md0.in‚5.3125 cpu‚25.043246929128426 proc.cpu.threads‚17 disk.nvme0n1.in‚ 353.99609375! disk.sda2.out‚187.88134765625! disk.sdb2.out‚187.88134765625 disk.sdb.in‚ 2.61328125 disk.sdb1.in‚0% disk./.usageGB‚2973.4401473999023 disk.md0.out‚ 181.55078125 disk.nvme1n1.in‚ 401.46875) proc.memory.availableMB‚ 11718.1328125# proc.memory.rssMB‚ 3044.60546875 network.sent‚ 2.420373e+06 network.recv‚616969 disk.sda2.in‚ 2.69921875 disk.sdb2.in‚ 2.61328125! disk.nvme1n1.out‚ 14189.265625* proc.memory.percent‚18.582797050476074µÎ뛶‚(:… ¥¤çÔâ¬Í‡ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚142.036' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.509 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05M5Ÿu·‚(:ê ¥¤çÔ鯈 network.sent‚ 2.425904e+06 disk.sdb2.out‚189.3623046875 disk.rbd0.out‚ 0.279296875! disk.nvme0n1.out‚ 8699.6015625 disk.dm-0.out‚ 23091.640625 disk.rbd0.in‚0.015625" disk.nvme1n1.out‚ 14392.0390625 disk.sdb1.out‚0 disk.sda.out‚189.3623046875 disk.rbd2.in‚0 disk.dm-0.in‚39015.23046875 disk.sda1.in‚0 disk.md0p1.in‚ 5.3203125 disk.nvme1n1.in‚ 15599.390625 disk.sda.in‚ 2.70703125" disk.nvme0n1.in‚23415.76953125 network.recv‚620330) disk./.usagePercent‚83.14210813603125 disk.md0.in‚ 5.3203125 disk.md0.out‚ 182.9765625 disk.sda2.in‚ 2.70703125 disk.sdb1.in‚0 disk.sda1.out‚0$ memory_percent‚28.86788845062256 cpu‚25.04301700046531 disk.rbd2.out‚ 0.279296875 disk.sdb2.in‚ 2.61328125 disk.sda2.out‚189.3623046875* proc.memory.percent‚18.582797050476074 proc.cpu.threads‚17% disk./.usageGB‚2973.5645065307617 disk.sdb.in‚ 2.61328125 disk.sdb.out‚189.3623046875 disk.md0p1.out‚ 182.9765625* proc.memory.availableMB‚11654.28515625# proc.memory.rssMB‚ 3044.60546875¡”Â2¸- _wandbruntime‚1755  _runtime‚1755š‘‰Ç¹Á & lr-AdamW/pg2‚0.0008234634281771179 trainer/global_step‚1449 % _timestamp‚1.7884657019612484e+09 _runtime‚1755.146373314 _step‚115 & lr-AdamW/pg1‚0.0008234634281771179–ÿ˺Š_step‚115 & lr-AdamW/pg1‚0.0008234634281771179 & lr-AdamW/pg2‚0.0008234634281771179 trainer/global_step‚1449 % _timestamp‚1.7884657019612484e+09 _runtime‚1755.146373314sÕ㙈2»- _wandbruntime‚1755  _runtime‚1755>‡ÇKÈ'¼ÂO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚61753678222848 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2877286968832 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66858221 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27480307 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 % _timestamp‚1.7884657019621067e+09 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16407586 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2929184171008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚58912009942057 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚58876642924544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚56042882172816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚61753929893376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚58876624921600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 _runtime‚1755.147163017 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39377914 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27480305 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26840945 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚62371646507520 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2929468499968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚58876642924544 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2877053301248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308269056 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 trainer/global_step‚1449 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚61753678222848 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚58876624921600 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16407584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 _step‚116 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚56042865133456±åéÌ'½ÆO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27480307 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 % _timestamp‚1.7884657019621067e+09 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16407586 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2929184171008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚58912009942057 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚58876642924544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚56042882172816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚61753929893376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚58876624921600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 _runtime‚1755.147163017 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39377914 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27480305 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26840945 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚62371646507520 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2929468499968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚58876642924544 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2877053301248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308269056 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 trainer/global_step‚1449 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚61753678222848 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚58876624921600 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16407584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 _step‚116 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚56042865133456 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚61753678222848 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2877286968832 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66858221t9f’a2¾- _wandbruntime‚1756  _runtime‚1756#t‹¿ÌM EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚58915928961536 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2879251008000 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚65344821429760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2879251008000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚62413354417664 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2871086258435 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2870857277151 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27498580 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2879017327104 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66876844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39404632 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚649 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308255744 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚62413330477568 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39404632 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚647 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚61794928285696 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27498582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27498580 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39378264 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚117 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚58951498788675 trainer/global_step‚1449 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚61795179969536 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66903214 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚58915928961536 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10440766 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315648 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27498582 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16418388 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16418386 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26858505 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300¢âL±Ö&ÀÐM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39378264 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26859154 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2931467012096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2931182696448 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚56080395490880 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2879017327104 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚58915910958592 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66876844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚61795179969536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66903214 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚58915910958592 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚56080412530240 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16418388 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16418386 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26858505 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚58915928961536 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2879251008000 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚65344821429760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2879251008000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚62413354417664 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2871086258435 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2870857277151 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27498580 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2879017327104 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66876844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39404632 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚649 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308255744 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚62413330477568 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39404632 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚647 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚61794928285696 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27498582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27498580 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39378264 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚117 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚58951498788675 trainer/global_step‚1449 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚61795179969536 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66903214 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚58915928961536 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10440766 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315648 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27498582 ODeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚63816195779072 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚60840926771200 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚670 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40715866 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69109923 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚67484819613696 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312463360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 trainer/global_step‚1499 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2967065809749 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28394057 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10781602 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚67485132077056 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚57919383004656 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _runtime‚1807.715320801 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚60840944774144 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28394057 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚63816447449600 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚60840944774144 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚63816447449600 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3029263127040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3028974603776ÒµÕä7ÏÆO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28394057 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚63816447449600 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚60840944774144 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚63816447449600 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3029263127040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3028974603776 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2975269007872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16947686 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚60886465853765 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _step‚120 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28394055 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚63816195779072 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚60840926771200 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚670 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40715866 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69109923 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚67484819613696 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312463360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 trainer/global_step‚1499 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2967065809749 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28394057 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10781602 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚67485132077056 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚57919383004656 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _runtime‚1807.715320801 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚60840944774144 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582x>j÷i2Ð- _wandbruntime‚1808  _runtime‚1808;G!{Ò&ÑÌM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69154916 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40716216 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚67528494173696 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚60925954700383 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69128546 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28412330 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3030965193728 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40742584 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚64497552920064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚63857697525760 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2968795317659 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2977233033728 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚60880212808192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 % _timestamp‚1.7884657554774737e+09 _step‚121 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69154916 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27746273 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10787787 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314547200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16958486 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 _runtime‚1808.663287259 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40716216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚60880230811136 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2977466714624 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚60880230811136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚63857697525760 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28412332 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10788466 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69128546 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27746954 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚681 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2977233033728 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2977466714624 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28412332 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚679 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚57956930401440 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40742584 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚64497552920064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚63857697525760 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2968795317659 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2977233033728 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚60880212808192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 % _timestamp‚1.7884657554774737e+09 _step‚121 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69154916 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27746273 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10787787 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314547200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16958486 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 _runtime‚1808.663287259 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40716216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚60880230811136 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2977466714624 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚60880230811136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚63857697525760 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28412332 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10788466 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69128546 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27746954 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚681 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2977233033728 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2977466714624 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28412332 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚679 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚57956930401440 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319yJn#2Ó- _wandbruntime‚1808  _runtime‚1808:êóÖUÔÏ /loss/FRt1/assignment_loss‚4.166198180480436 /loss/FRt2/assignment_loss‚4.510910297961945 2loss/SRqqt1/assignment_loss‚2.2413664134624214 0loss/FBt2/assignment_loss‚0.4928246353837577 /loss/FRt3/detection_loss‚1.3630278863944114 /loss/FRt4/detection_loss‚2.9461237212022144 0loss/SRqqt1/detection_loss‚1.966784321743509 1loss/SRqqt3/detection_loss‚0.8034033707373744 .loss/FBt4/detection_loss‚2.849698516038748 $loss/total_loss‚49.1052879194407 % _timestamp‚1.7884657554798942e+09 2loss/SRqqt4/assignment_loss‚2.0255267161589403 epoch‚1 trainer/global_step‚1499 _step‚122 /loss/FRt4/assignment_loss‚4.042213602860769 1loss/FBt3/assignment_loss‚0.18559828761992458 1loss/FBt4/assignment_loss‚0.23511651204893708 .loss/FRt1/detection_loss‚2.476551563225009 .loss/FRt2/detection_loss‚2.290282263400707 1loss/SRqqt4/detection_loss‚2.2724451262217302 .loss/FBt1/detection_loss‚2.338213162762778 1loss/SRqqt2/detection_loss‚2.6845623707329787 .loss/FBt2/detection_loss‚2.911336140199141 /loss/FBt3/detection_loss‚0.6055084708865347 _runtime‚1808.664708606 /loss/FRt3/assignment_loss‚1.709166406840086 1loss/SRqqt2/assignment_loss‚1.999607847796546 2loss/SRqqt3/assignment_loss‚1.1867595113158882 0loss/FBt1/assignment_loss‚0.8020625939659141ȳ¸YÕÓ epoch‚1 trainer/global_step‚1499 _step‚122 /loss/FRt4/assignment_loss‚4.042213602860769 1loss/FBt3/assignment_loss‚0.18559828761992458 1loss/FBt4/assignment_loss‚0.23511651204893708 .loss/FRt1/detection_loss‚2.476551563225009 .loss/FRt2/detection_loss‚2.290282263400707 1loss/SRqqt4/detection_loss‚2.2724451262217302 .loss/FBt1/detection_loss‚2.338213162762778 1loss/SRqqt2/detection_loss‚2.6845623707329787 .loss/FBt2/detection_loss‚2.911336140199141 /loss/FBt3/detection_loss‚0.6055084708865347 _runtime‚1808.664708606 /loss/FRt3/assignment_loss‚1.709166406840086 1loss/SRqqt2/assignment_loss‚1.999607847796546 2loss/SRqqt3/assignment_loss‚1.1867595113158882 0loss/FBt1/assignment_loss‚0.8020625939659141 /loss/FRt1/assignment_loss‚4.166198180480436 /loss/FRt2/assignment_loss‚4.510910297961945 2loss/SRqqt1/assignment_loss‚2.2413664134624214 0loss/FBt2/assignment_loss‚0.4928246353837577 /loss/FRt3/detection_loss‚1.3630278863944114 /loss/FRt4/detection_loss‚2.9461237212022144 0loss/SRqqt1/detection_loss‚1.966784321743509 1loss/SRqqt3/detection_loss‚0.8034033707373744 .loss/FBt4/detection_loss‚2.849698516038748 $loss/total_loss‚49.1052879194407 % _timestamp‚1.7884657554798942e+09 2loss/SRqqt4/assignment_loss‚2.0255267161589403z8†â·’Ö‚(:‡ á¤çÔÇöˆ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚117.525' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.38125 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª¬Jþwׂ(:ì á¤çÔ€š×‰ disk.sdb.out‚195.50048828125 disk.sdb1.out‚0 disk.sda1.in‚0 disk.rbd2.out‚ 0.279296875 disk.sda.in‚ 3.52734375 disk.sda.out‚195.50048828125* proc.memory.percent‚18.582797050476074 disk.sdb.in‚ 3.40234375# disk.nvme0n1.in‚165714.67578125 disk.md0.in‚ 6.9296875 disk.md0.out‚ 188.8984375 disk.rbd2.in‚0 cpu‚25.039285356910188$ disk./.usageGB‚2973.992935180664 disk.dm-0.out‚ 23532.5625 disk.sda1.out‚0# disk.nvme1n1.in‚130934.57421875! disk.nvme1n1.out‚ 14831.140625$ memory_percent‚30.85494041442871 network.sent‚ 2.547499e+06 disk.sda2.in‚ 3.52734375 disk.rbd0.out‚ 0.279296875 network.recv‚638639) disk./.usagePercent‚83.15408718039386 disk.nvme0n1.out‚ 8701.421875 disk.sdb2.in‚ 3.40234375! disk.sdb2.out‚195.50048828125 disk.sdb1.in‚0 disk.dm-0.in‚ 296649.984375! disk.sda2.out‚195.50048828125 disk.md0p1.out‚ 188.8984375 disk.rbd0.in‚0.015625) proc.memory.availableMB‚ 11328.7265625 proc.cpu.threads‚17 disk.md0p1.in‚ 6.9296875# proc.memory.rssMB‚ 3044.60546875B[GbØ‚(:‡ ð¤çÔ©©Ä‡ gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚141.427' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.35675 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemor!pí^0yErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªü­¿uÙ‚(:ê ð¤çÔ›Ðú‡( disk./.usagePercent‚83.1540773676249 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 0.279296875" disk.nvme1n1.out‚ 15092.9765625 disk.dm-0.in‚315204.92578125 disk.dm-0.out‚23794.5 disk.md0p1.out‚ 190.99609375 disk.sdb2.in‚ 3.40234375) proc.memory.availableMB‚ 11328.9453125! disk.nvme0n1.out‚ 8701.5234375 disk.md0p1.in‚ 6.93359375 network.sent‚ 2.65689e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚3.53125! disk.sdb2.out‚197.65380859375 disk.sdb.out‚197.65380859375* proc.memory.percent‚18.582797050476074 disk.sdb1.in‚0 disk.sda.in‚3.53125! disk.sda2.out‚197.65380859375 disk.md0.out‚ 190.99609375% memory_percent‚30.853605270385742 cpu‚25.039150249056963 proc.cpu.threads‚17 network.recv‚643912# disk.nvme1n1.in‚134760.32421875 disk.sdb.in‚ 3.40234375# proc.memory.rssMB‚ 3044.60546875% disk./.usageGB‚2973.9925842285156 disk.sda.out‚197.65380859375" disk.nvme0n1.in‚180444.6015625 disk.rbd0.out‚ 0.279296875 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.md0.in‚ 6.93359375ÐćܜÚ‚(:‘ ÿ¤çÔÌ̃† gpu.0.gpu‚10.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gHâÙuÛ‚(:ê ÿ¤çÔÆüƒ‡ disk.rbd0.out‚ 0.279296875 disk.md0p1.in‚ 7.01171875 disk.md0.out‚ 192.42578125 disk.dm-0.out‚ 23794.6640625 proc.cpu.threads‚17% disk./.usageGB‚2973.9981002807617 disk.sdb2.in‚3.453125 disk.sdb2.out‚ 199.150390625 disk.rbd2.out‚ 0.279296875 disk.dm-0.in‚332361.47265625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚ 199.150390625% memory_percent‚30.856657028198242* proc.memory.percent‚18.582797050476074 network.sent‚ 2.659878e+06 disk.rbd0.in‚0.015625 disk.nvme1n1.in‚ 138179.3125 disk.sdb1.out‚0 disk.sda.out‚ 199.150390625" disk.nvme0n1.out‚ 8701.57421875) proc.memory.availableMB‚ 11328.4453125 cpu‚25.03886355766892 disk.sda2.in‚ 3.55859375 network.recv‚647578) disk./.usagePercent‚83.15423159875435 disk.rbd2.in‚0 disk.sdb.in‚3.453125 disk.sdb.out‚ 199.150390625# disk.nvme0n1.in‚194182.16015625 disk.md0p1.out‚ 192.42578125 disk.sdb1.in‚0# proc.memory.rssMB‚ 3044.60546875 disk.md0.in‚ 7.01171875# disk.nvme1n1.out‚15093.08984375 disk.sda.in‚ 3.55859375à m’Ü‚(:‡ Ž¥çÔÐÏ“† gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚142.653' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.66325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z—||Ý‚(:ñ Ž¥çÔò¡‡ disk.md0p1.out‚ 193.7578125 disk.md0.out‚ 193.7578125# disk.nvme1n1.out‚15304.41015625% memory_percent‚30.855870246887207 cpu‚25.0388134029028 network.recv‚652125 disk.sda.in‚ 3.55859375" disk.nvme0n1.out‚ 8701.89453125) disk./.usagePercent‚83.15420418699759! disk.sdb2.out‚200.55419921875 disk.rbd0.out‚ 0.279296875 disk.rbd2.out‚ 0.279296875* proc.memory.percent‚18.582797050476074 disk.md0.in‚ 7.01171875 network.sent‚ 2.66634e+06 disk.sda1.in‚0 disk.sda2.in‚ 3.55859375 disk.sdb1.out‚0 disk.dm-0.in‚ 346347.078125 disk.sdb.in‚3.453125* proc.memory.availableMB‚11328.57421875 proc.cpu.threads‚17 disk.sdb2.in‚3.453125 disk.sda1.out‚0 disk.sdb1.in‚0 disk.sda.out‚200.55419921875# proc.memory.rssMB‚ 3044.60546875 disk.rbd2.in‚0# disk.nvme0n1.in‚205616.55078125% disk./.usageGB‚2973.9971199035645! disk.sda2.out‚200.55419921875 disk.md0p1.in‚ 7.01171875 disk.rbd0.in‚0.015625# disk.nvme1n1.in‚140730.52734375 disk.sdb.out‚200.55419921875 disk.dm-0.out‚ 24006.3046875Ø%|2Þ- _wandbruntime‚1861  _runtime‚1861Á/JnÅß¿ trainer/global_step‚1549 % _timestamp‚1.7884658080952697e+09 _runtime‚1861.280237893 _step‚123 % lr-AdamW/pg1‚0.000822468084133655 % lr-AdamW/pg2‚0.000822468084133655%',ÒÉàà % lr-AdamW/pg1‚0.000822468084133655 % lr-AdamW/pg2‚0.000822468084133655 trainer/global_step‚1549 % _timestamp‚1.7884658080952697e+09 _runtime‚1861.280237893 _step‚123{ûе2á- _wandbruntime‚1861  _runtime‚1861Âæ|ÃÈ'âÂO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28617575 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3065003850257 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29307807 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11129115 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42027476 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚62805246623744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17487784 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292717568 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29307807 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚65878965005824 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _runtime‚1861.281396368 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3073484714496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚65878713335296 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚62805228620800 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚1549 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17487786 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚65878713335296 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3073718382080 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71335281 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚65878965005824 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚66540067452416 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104/Tï?Ì'ãÆO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28617575 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3065003850257 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29307807 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11129115 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42027476 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚62805246623744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17487784 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292717568 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29307807 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚65878965005824 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _runtime‚1861.281396368 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3073484714496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚65878713335296 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚62805228620800 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚1549 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17487786 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚65878713335296 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3073718382080 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71335281 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚65878965005824 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚66540067452416 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11136132 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚62844532660736 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚62900164591447 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 _step‚125 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29326082 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71406618 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚69712820550144 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚676 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚66581727482368 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71406618 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42054168 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17498586 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42080536 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3075682421248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚62900410612091 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28635396 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚62844514657792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3131069127680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.smalæÈL° l_pool.freed‚ 3066733358167 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17498588 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3075448740352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚59833431233280 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1549 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28634718 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚62844514657792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42080536 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚69712508100096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0»uÄÁÖ&æÐM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚62900164591447 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 _step‚125 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29326082 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71406618 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚69712820550144 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚676 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚66581727482368 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71406618 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42054168 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17498586 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42080536 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3075682421248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚62900410612091 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28635396 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚62844514657792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3131069127680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3066733358167 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17498588 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3075448740352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚59833431233280 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1549 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28634718 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚62844514657792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42080536 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚69712508100096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚62844532660736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚59833448272640 _runtime‚1862.247113229 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚65920215081984 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71380248 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11136132 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚62844532660736 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712}c"‹2ç- _wandbruntime‚1862  _runtime‚18623ƒŒUVèÐ 0loss/FRt1/assignment_loss‚3.8132422525774348 1loss/SRqqt1/detection_loss‚2.2616702769253707 0loss/FBt4/assignment_loss‚0.3596280825856536 /loss/FRt3/detection_loss‚1.4822944826313427 .loss/FBt1/detection_loss‚2.017188748892616 trainer/global_step‚1549 $ _timestamp‚1.788465809064458e+09 0loss/FRt4/assignment_loss‚5.1764372744058305 /loss/FRt1/detection_loss‚2.4358568448912012 0loss/SRqqt2/detection_loss‚2.431380183310122 1loss/SRqqt2/assignment_loss‚2.056784418789116 2loss/SRqqt4/assignment_loss‚2.2891966906877665 /loss/FRt4/detection_loss‚2.6503658906409613 .loss/FBt2/detection_loss‚3.320060610771179 _step‚126 .loss/FRt2/detection_loss‚2.444717453076289 0loss/SRqqt4/detection_loss‚2.879757720690507 epoch‚1 _runtime‚1862.249195029 /loss/FRt2/assignment_loss‚4.288015442016797 0loss/SRqqt1/assignment_loss‚1.64643630732757 0loss/FBt2/assignment_loss‚0.4219840483000957 /loss/FBt3/detection_loss‚0.8109315123217374 .loss/FBt4/detection_loss‚2.356662141425269 0loss/FRt3/assignment_loss‚1.5130194229739053 2loss/SRqqt3/assignment_loss‚1.2565524073279635 0loss/FBt1/assignment_loss‚0.3398778519099123 1loss/FBt3/assignment_loss‚0.16118377165592884 1loss/SRqqt3/detection_loss‚1.0074307771057498 &loss/total_loss‚49.420674613240315ÿÏ)ŸZéÔ 0loss/FRt3/assignment_loss‚1.5130194229739053 2loss/SRqqt3/assignment_loss‚1.2565524073279635 0loss/FBt1/assignment_loss‚0.3398778519099123 1loss/FBt3/assignment_loss‚0.16118377165592884 1loss/SRqqt3/detection_loss‚1.0074307771057498 &loss/total_loss‚49.420674613240315 0loss/FRt1/assignment_loss‚3.8132422525774348 1loss/SRqqt1/detection_loss‚2.2616702769253707 0loss/FBt4/assignment_loss‚0.3596280825856536 /loss/FRt3/detection_loss‚1.4822944826313427 .loss/FBt1/detection_loss‚2.017188748892616 trainer/global_step‚1549 $ _timestamp‚1.788465809064458e+09 0loss/FRt4/assignment_loss‚5.1764372744058305 /loss/FRt1/detection_loss‚2.4358568448912012 0loss/SRqqt2/detection_loss‚2.431380183310122 1loss/SRqqt2/assignment_loss‚2.056784418789116 2loss/SRqqt4/assignment_loss‚2.2891966906877665 /loss/FRt4/detection_loss‚2.6503658906409613 .loss/FBt2/detection_loss‚3.320060610771179 _step‚126 .loss/FRt2/detection_loss‚2.444717453076289 0loss/SRqqt4/detection_loss‚2.879757720690507 epoch‚1 _runtime‚1862.249195029 /loss/FRt2/assignment_loss‚4.288015442016797 0loss/SRqqt1/assignment_loss‚1.64643630732757 0loss/FBt2/assignment_loss‚0.4219840483000957 /loss/FBt3/detection_loss‚0.8109315123217374 .loss/FBt4/detection_loss‚2.356662141425269~lLÉ]ê‚(:… ¥çÔ’žß… gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚139.868' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.967 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 µìhë‚(:Ý ¥çÔËšþ† disk.sdb.in‚3.453125 disk.sda2.out‚ 202.3984375 disk.md0p1.out‚ 195.546875* proc.memory.percent‚18.582797050476074 disk.md0p1.in‚ 7.01171875! disk.nvme1n1.in‚ 142351.671875% memory_percent‚30.857110023498535 disk.sdb1.in‚0 disk.sda.in‚ 3.55859375# proc.memory.rssMB‚ 3044.60546875 cpu‚25.03864684673692$ disk./.usageGB‚2974.123519897461 disk.sdb.out‚ 202.3984375 disk.sda1.out‚0 disk.sdb2.in‚3.453125 disk.rbd0.out‚ 0.291015625 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.sda2.in‚ 3.55859375 network.recv‚656488) disk./.usagePercent‚83.15773838372944 disk.sdb2.out‚ 202.3984375# disk.nvme0n1.in‚215531.00390625 disk.nvme0n1.out‚ 8701.9375 disk.sda1.in‚0 proc.cpu.threads‚17 network.sent‚ 2.672265e+06 disk.md0.out‚ 195.546875 disk.rbd0.in‚0.015625 disk.sda.out‚ 202.3984375 disk.dm-0.in‚357882.67578125 disk.dm-0.out‚24094.41015625* proc.memory.availableMB‚11328.37109375 disk.md0.in‚ 7.01171875 disk.rbd2.out‚ 0.291015625# disk.nvme1n1.out‚15392.47265625oß©’ì‚(:‡ ¬¥çÔ×ÍÕ… gpu.0.gpu‚42.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚143.595' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.89875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vTÓyí‚(:î ¬¥çÔäÇå† disk.sdb.out‚203.86474609375 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 0.291015625 disk.sdb.in‚3.453125 disk.sda.in‚ 3.55859375 disk.md0.out‚ 196.953125 disk.dm-0.out‚ 24278.3203125 disk.md0p1.in‚ 7.01171875% memory_percent‚30.855727195739746* proc.memory.percent‚18.582797050476074( disk./.usagePercent‚83.1612340760092% disk./.usageGB‚2974.2485427856445 disk.md0.in‚ 7.01171875" disk.nvme1n1.in‚143936.9765625 disk.sdb1.out‚0 disk.nvme0n1.in‚ 225184.84375 disk.dm-0.in‚369121.9765625# proc.memory.rssMB‚ 3044.60546875 disk.rbd2.in‚0 disk.sdb1.in‚0 disk.md0p1.out‚ 196.953125 proc.cpu.threads‚17 network.sent‚ 2.779646e+06 network.recv‚662056! disk.sdb2.out‚203.86474609375 disk.sdb2.in‚3.453125# disk.nvme1n1.out‚15576.26953125 disk.sda.out‚203.86474609375 disk.sda1.out‚0 disk.sda2.in‚ 3.55859375! disk.sda2.out‚203.86474609375* proc.memory.availableMB‚11328.59765625 disk.rbd0.out‚ 0.353515625" disk.nvme0n1.out‚ 8702.05078125 disk.sda1.in‚0 cpu‚25.038480594237516x»–±’î‚(:‡ »¥çÔ§­×… gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚117.193' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.29825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|aò¼cï‚(:Ø »¥çÔç°×† disk.md0.out‚ 198.4453125 disk.sda1.in‚0 disk.sdb1.out‚0 disk.dm-0.in‚380524.7421875 disk.md0p1.in‚ 7.01171875 cpu‚25.038314845191923 disk.rbd2.out‚ 0.353515625 network.recv‚665635 disk.sda.in‚ 3.55859375 disk.nvme0n1.in‚ 235241.9375 disk.md0p1.out‚ 198.4453125 disk.md0.in‚ 7.01171875 disk.rbd0.in‚0.015625 disk.sdb.in‚3.453125 disk.sdb1.in‚0 disk.sda2.in‚ 3.55859375 disk.sda1.out‚0# disk.nvme1n1.out‚15758.70703125 network.sent‚ 2.784897e+06) disk./.usagePercent‚83.14638660995625% disk./.usageGB‚2973.7175254821777 disk.rbd2.in‚0 disk.sdb.out‚ 205.421875% memory_percent‚30.855751037597656 disk.rbd0.out‚ 0.353515625 disk.sda.out‚ 205.421875# proc.memory.rssMB‚ 3044.60546875 disk.sdb2.in‚3.453125 disk.sdb2.out‚ 205.421875" disk.nvme0n1.out‚ 8702.18359375 disk.sda2.out‚ 205.421875* proc.memory.percent‚18.582797050476074 proc.cpu.threads‚17" disk.nvme1n1.in‚145282.8046875 disk.dm-0.out‚ 24460.890625' proc.memory.availableMB‚ 11328.59375!¤2ð- _wandbruntime‚1912  _runtime‚1912A Äñ¾ $ _timestamp‚1.788465859186752e+09 _runtime‚1912.371683914 _step‚127 % lr-AdamW/pg1‚0.000821842115151383 % lr-AdamW/pg2‚0.000821842115151383 trainer/global_step‚1599Ð:ßJÈò % lr-AdamW/pg1‚0.000821842115151383 % lr-AdamW/pg2‚0.000821842115151383 trainer/global_step‚1599 $ _timestamp‚1.788465859186752e+09 _runtime‚1912.371683914 _step‚127ùV‹î2ó- _wandbruntime‚1912  _runtime‚1912_ßPqÈ'ôÂO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30221555 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚64769530470400 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3162941890765 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚67941482562048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3171700421120 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚707 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3171934088704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18027884 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚64769548473344 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚68624242014720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3228518680064 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43391770 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚67941230891520 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30221557 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29507602 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚71853075255296 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚71852760694784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3228809300480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11479009 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 % _timestamp‚1.7884658591877685e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚73586983 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30221557 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚67941230891520 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314560512 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3171934088704 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29506893 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3162712909689 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚68624265954816 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚61672435786416 trainer/global_step‚1599 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43365428 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73613327 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43391770 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚64835377677181 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚64769548473344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11479716 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43365428 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30221555 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18027886 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290620416‰=¼vqõÇO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11479009 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 % _timestamp‚1.7884658591877685e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚73586983 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30221557 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚67941230891520 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314560512 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3171934088704 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29506893 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3162712909689 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚68624265954816 _LDeviceStatsMonitor.on_train_bEîÍ\atch_start/requested_bytes.large_pool.allocated‚61672435786416 trainer/global_step‚1599 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43365428 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73613327 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43391770 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚64835377677181 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚64769548473344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11479716 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43365428 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30221555 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18027886 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290620416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30221555 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚64769530470400 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3162941890765 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚67941482562048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3171700421120 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚707 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3171934088704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18027884 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚64769548473344 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚68624242014720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3228518680064 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43391770 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚67941230891520 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30221557 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29507602 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚71853075255296 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚71852760694784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3228809300480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0€µÄ@#2ö- _wandbruntime‚1913  _runtime‚1913æ{ZÒ&÷ÌM bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚68665949924864 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43392120 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3230507348480 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18038686 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚64808816507392 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43418488 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚61709949104480 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚68665925984768 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29525080 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚73631950 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3230804247040 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296898560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30239832 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30239830 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3164900379959 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚61709966143840 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30239830 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3164671398675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚1913.325960301 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3173898127872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚675 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚64808834510336 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚64874866523799 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚67982732638208 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚64808816507392 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11485717 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚67982480954368 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73658320 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚677 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚71896754171904 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 trainer/global_step‚1599 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚73631950 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11486392 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚64874620503155 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3173898127872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43418488 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29524403 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18038688×TSA×&øÑM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚675 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚64808834510336 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚64874866523799 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚67982732638208 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚64808816507392 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11485717 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚67982480954368 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73658320 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚677 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚71896754171904 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 trainer/global_step‚1599 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚73631950 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11486392 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚64874620503155 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3173898127872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43418488 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29524403 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18038688 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚68665949924864 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43392120 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3230507348480 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18038686 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚64808816507392 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43418488 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚61709949104480 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚68665925984768 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29525080 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚73631950 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3230804247040 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296898560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30239832 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30239830 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3164900379959 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚61709966143840 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30239830 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3164671398675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚1913.325960301 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3173898127872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2Ph͘2ù- _wandbruntime‚1913  _runtime‚19131ö¦GTúÎ 1loss/SRqqt4/assignment_loss‚2.077829024003398 1loss/FBt4/assignment_loss‚0.40672437825964397 _step‚130 .loss/FBt2/detection_loss‚2.402746954384972 -loss/FRt1/detection_loss‚3.24746439854304 0loss/SRqqt2/assignment_loss‚2.00050104212235 1loss/FBt1/assignment_loss‚0.20328994592027622 /loss/FBt1/detection_loss‚2.7898712356885276 _runtime‚1913.326962755 0loss/FRt3/assignment_loss‚1.4736672453582287 1loss/SRqqt1/assignment_loss‚1.871566544740628 /loss/FRt2/detection_loss‚2.5715026488670936 0loss/SRqqt3/detection_loss‚0.819700646014618 /loss/FBt4/detection_loss‚3.2034320169024997 epoch‚1 /loss/FRt3/detection_loss‚1.6119165696727578 /loss/SRqqt2/detection_loss‚2.31025573961875 trainer/global_step‚1599 1loss/FBt2/assignment_loss‚0.14197888921069748 /loss/FRt4/detection_loss‚2.7545896490414936 % _timestamp‚1.7884658601422713e+09 %loss/total_loss‚48.89465463562854 /loss/FRt2/assignment_loss‚4.416445704606863 0loss/FRt4/assignment_loss‚4.2827274084091185 2loss/SRqqt3/assignment_loss‚1.2448156877071597 1loss/FBt3/assignment_loss‚0.16446186824492834 0loss/SRqqt1/detection_loss‚2.183620948057908 0loss/SRqqt4/detection_loss‚2.463502520515073 /loss/FRt1/assignment_loss‚3.435010313987732 .loss/FBt3/detection_loss‚0.817033255750786 ôiYûÓ .loss/FBt3/detection_loss‚0.817033255750786 1loss/SRqqt4/assignment_loss‚2.077829024003398 1loss/FBt4/assignment_loss‚0.40672437825964397 _step‚130 .loss/FBt2/detection_loss‚2.402746954384972 -loss/FRt1/detection_loss‚3.24746439854304 0loss/SRqqt2/assignment_loss‚2.00050104212235 1loss/FBt1/assignment_loss‚0.20328994592027622 /loss/FBt1/detection_loss‚2.7898712356885276 _runtime‚1913.326962755 0loss/FRt3/assignment_loss‚1.4736672453582287 1loss/SRqqt1/assignment_loss‚1.871566544740628 /loss/FRt2/detection_loss‚2.5715026488670936 0loss/SRqqt3/detection_loss‚0.819700646014618 /loss/FBt4/detection_loss‚3.2034320169024997 epoch‚1 /loss/FRt3/detection_loss‚1.6119165696727578 /loss/SRqqt2/detection_loss‚2.31025573961875 trainer/global_step‚1599 1loss/FBt2/assignment_loss‚0.14197888921069748 /loss/FRt4/detection_loss‚2.7545896490414936 % _timestamp‚1.7884658601422713e+09 %loss/total_loss‚48.89465463562854 /loss/FRt2/assignment_loss‚4.416445704606863 0loss/FRt4/assignment_loss‚4.2827274084091185 2loss/SRqqt3/assignment_loss‚1.2448156877071597 1loss/FBt3/assignment_loss‚0.16446186824492834 0loss/SRqqt1/detection_loss‚2.183620948057908 0loss/SRqqt4/detection_loss‚2.463502520515073 /loss/FRt1/assignment_loss‚3.435010313987732‚ö÷æî’ü‚(:‡ Ê¥çÔÕ¼½† gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚133.349' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.33725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07PÃÄsý‚(:è Ê¥çÔå½ï† network.sent‚ 2.791227e+06 disk.sdb.in‚ 3.49609375 cpu‚25.038143521830897 proc.cpu.threads‚17 disk.md0.out‚ 200.48828125 disk.sdb1.in‚0 network.recv‚670444" disk.nvme0n1.out‚ 8702.33984375 disk.nvme1n1.in‚ 146232.5625# disk.nvme1n1.out‚15759.66796875 disk.sdb1.out‚0 disk.sda.in‚ 3.61328125% memory_percent‚30.858945846557617* proc.memory.percent‚18.582797050476074) proc.memory.availableMB‚ 11328.0703125) disk./.usagePercent‚83.15200591343056 disk.sda2.in‚ 3.61328125 disk.sda2.out‚ 207.541015625 disk.sdb2.in‚ 3.49609375 disk.rbd2.in‚0 disk.sda1.in‚0 disk.md0.in‚7.109375 disk.rbd0.out‚ 0.4326171875# proc.memory.rssMB‚ 3044.60546875# disk./.usageGB‚2973.91849899292 disk.dm-0.in‚389822.8046875 disk.dm-0.out‚ 24462.0078125 disk.sda1.out‚0 disk.md0p1.out‚ 200.48828125 disk.sdb2.out‚ 207.541015625 disk.rbd2.out‚ 0.353515625 disk.sdb.out‚ 207.541015625" disk.nvme0n1.in‚243590.2421875 disk.md0p1.in‚7.109375 disk.rbd0.in‚0.015625 disk.sda.out‚ 207.541015625U¨½¹’þ‚(:‡ Ù¥çÔŒ¸÷… gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚139.251' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.81275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c¿Ùÿ‚ï ·u(:ð Ù¥çÔÃÂý† disk.sdb2.in‚ 3.49609375 disk.sdb2.out‚208.9326171875 disk.md0.out‚ 201.82421875 disk.sdb1.in‚0# disk.nvme0n1.in‚252660.36328125) proc.memory.availableMB‚ 11328.5546875* proc.memory.percent‚18.582797050476074 disk.rbd0.in‚0.015625 disk.sdb.in‚ 3.49609375 disk.dm-0.in‚ 400862.90625% memory_percent‚30.855989456176758 proc.cpu.threads‚17 disk.md0.in‚7.109375" disk.nvme1n1.in‚148202.2421875 disk.sdb1.out‚0 disk.sda2.out‚208.9326171875 cpu‚25.038103958588668 network.sent‚ 2.897933e+06( disk./.usagePercent‚83.1682005020042% disk./.usageGB‚2974.4976959228516" disk.nvme1n1.out‚ 15959.2421875 disk.sda2.in‚ 3.61328125 disk.md0p1.out‚ 201.82421875 disk.sda1.in‚0 disk.sda.in‚ 3.61328125 disk.sda.out‚208.9326171875" disk.nvme0n1.out‚ 8783.09765625 disk.dm-0.out‚24742.33984375 network.recv‚675437 disk.rbd0.out‚ 0.4326171875 disk.rbd2.in‚0 disk.md0p1.in‚7.109375# proc.memory.rssMB‚ 3044.60546875 disk.rbd2.out‚ 0.4326171875 disk.sdb.out‚208.9326171875 disk.sda1.out‚0ý œ€‚(:‘ è¥çÔ¤÷‡ gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–Ï-”u‚(:ê è¥çÔªüö‡# proc.memory.rssMB‚ 3044.60546875* proc.memory.percent‚18.582797050476074 proc.cpu.threads‚17 disk.md0.out‚ 203.109375 disk.sda2.in‚ 3.61328125 disk.sda1.out‚0$ memory_percent‚30.85470199584961 disk.rbd0.out‚ 0.49560546875# disk.nvme1n1.in‚149358.71484375 disk.dm-0.out‚24944.68359375 disk.sda1.in‚0( disk./.usagePercent‚83.1690870643472% disk./.usageGB‚2974.5294036865234 disk.nvme1n1.out‚ 16158.90625 disk.sdb2.in‚ 3.49609375! disk.sdb2.out‚210.27978515625 disk.md0.in‚7.109375 disk.sdb1.in‚0 disk.sdb1.out‚0" disk.nvme0n1.out‚ 8785.77734375 disk.dm-0.in‚ 410451.25 network.sent‚ 2.905512e+06 disk.rbd2.in‚0! disk.sda2.out‚210.27978515625 disk.md0p1.out‚ 203.109375 disk.sdb.in‚ 3.49609375 disk.sdb.out‚210.27978515625 disk.sda.out‚210.27978515625 disk.md0p1.in‚7.109375( proc.memory.availableMB‚ 11328.765625 cpu‚25.03804275534826 network.recv‚679851 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 0.4326171875 disk.sda.in‚ 3.61328125# disk.nvme0n1.in‚261092.53515625?Å̘‘‚‚(:† ÷¥çÔ둾… gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚116.306' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.0765 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üÞèhƒ‚(:Ý ÷¥çÔŠÎë† disk.nvme0n1.in‚297318 disk.rbd2.out‚ 0.49560546875 disk.nvme1n1.out‚16279 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0 disk.md0p1.in‚ 7.11328125# disk.nvme1n1.in‚174334.91015625 disk.sdb2.in‚ 3.49609375 disk.rbd2.in‚0 disk.sdb.out‚ 211.755859375 disk.sda2.out‚ 211.755859375 disk.sdb2.out‚ 211.755859375$ memory_percent‚30.85768222808838# proc.memory.rssMB‚ 3044.60546875 disk.dm-0.out‚25152.73828125 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 0.49560546875 disk.sda.in‚ 3.6171875 disk.sda.out‚ 211.755859375 cpu‚25.037910877881668 proc.cpu.threads‚17 disk.sdb.in‚ 3.49609375 disk.dm-0.in‚471649.98828125 network.sent‚ 2.91076e+06 disk.sda1.out‚0 disk.md0p1.out‚ 204.53125* proc.memory.availableMB‚11328.27734375* proc.memory.percent‚18.582797050476074 network.recv‚683743% disk./.usageGB‚2974.6230125427246" disk.nvme0n1.out‚ 8873.73828125 disk.sda2.in‚ 3.6171875) disk./.usagePercent‚83.17170440714492 disk.md0.in‚ 7.11328125 disk.md0.out‚ 204.53125ð]«Û2„- _wandbruntime‚1966  _runtime‚1966ØQ¥ÌÇ…Á & lr-AdamW/pg1‚0.0008211307828578235 & lr-AdamW/pg2‚0.0008211307828578235 trainer/global_step‚1649 % _timestamp‚1.7884659134810095e+09 _runtime‚1966.666103841 _step‚131½AüUÌ†Æ _runtime‚1966.666103841 _step‚131 & lr-AdamW/pg1‚0.0008211307828578235 & lr-AdamW/pg2‚0.0008211307828578235 trainer/global_step‚1649 % _timestamp‚1.7884659134810095e+09ƒ„*2‘2‡- _wandbruntime‚1966  _runtime‚1966 ÅoµÈ'ˆÂO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11824024 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚70708464457216 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44703380 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚66733832320000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30392008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3269916127744 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3269916127744 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 320851968 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 296911872 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44729722 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3260650950197 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚70004000118272 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚709 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚66733850322944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚66809833588889 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚70003748447744 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11824733 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚70708440517120 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚70003748447744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18567984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3270149795328 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18567986 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75838685 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚66733832320000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31135307 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚74036736046592 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30392719 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚63548953657616 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096o•õ]Í'‰ÇO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30392719 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚63548953657616 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11824024 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚70708464457216 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44703380 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚66733832320000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30392008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3269916127744 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3269916127744 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 320851968 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 296911872 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44729722 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3260650950197 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚70004000118272 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚709 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚66733850322944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚66809833588889 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚70003748447744 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11824733 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚70708440517120 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚70003748447744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18567984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3270149795328 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18567986 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75838685 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚66733832320000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31135307 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚74036736046592 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273„ÖÕý2Š- _wandbruntime‚1967  _runtime‚19670„+ûh‹ÌM R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚70044998510592 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44730072 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚74080728665600 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75883652 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7884659145827277e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚63586466975680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18578788 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚74080420409856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3330580238336 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚70045250194432 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3272113834496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1967.769671646 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3262838420467 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3271880153600 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3271880153600 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31153580 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11831975 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75883652 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚66773118356992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44730072 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚66849322435507 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31153582 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.smal¿²NQj l_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 trainer/global_step‚1649 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚70045250194432 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75910022 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30410081 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚75910022 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31153582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18578786 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚66849076414863 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3272113834496 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44756440 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3330295922688 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚682 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582SûEš×&ŒÑM D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1967.769671646 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3262838420467 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3271880153600 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3271880153600 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31153580 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11831975 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75883652 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚66773118356992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44730072 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚66849322435507 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31153582 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 trainer/global_step‚1649 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚70045250194432 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75910022 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30410081 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚75910022 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31153582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18578786 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚66849076414863 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3272113834496 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44756440 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3330295922688 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚682 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚70044998510592 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44730072 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚74080728665600 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75883652 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7884659145827277e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚63586466975680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18578788 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚74080420409856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3330580238336 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚70045250194432 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3272113834496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582…-tç2- _wandbruntime‚1967  _runtime‚1967¹TÆiSŽÍ /loss/FRt3/assignment_loss‚1.695600921839056 /loss/FRt4/assignment_loss‚5.253882769382361 .loss/FRt3/detection_loss‚1.659816100353926 1loss/FBt4/assignment_loss‚0.36560843620100014 .loss/FRt1/assignment_loss‚3.98461183273431 0loss/SRqqt1/detection_loss‚2.314463702901717 0loss/SRqqt4/detection_loss‚2.782877641645345 /loss/FBt3/detection_loss‚0.7541383078477035 1loss/SRqqt2/detection_loss‚2.3000048220157625 .loss/FBt2/detection_loss‚2.797730489419057 trainer/global_step‚1649 0loss/FBt1/assignment_loss‚0.5140430081466344 /loss/FBt1/detection_loss‚2.4390590245073493 .loss/FBt4/detection_loss‚2.244276402518153 $loss/total_loss‚49.0872190141844 .loss/FRt1/detection_loss‚2.868782849925937 -loss/FRt2/detection_loss‚2.34501641295677 0loss/FRt2/assignment_loss‚3.8175286110057387 2loss/SRqqt3/assignment_loss‚1.0733579871044538 2loss/SRqqt2/assignment_loss‚2.0663852052763105 /loss/FRt4/detection_loss‚2.8263943990071616 1loss/SRqqt4/assignment_loss‚2.399992225522345 0loss/FBt3/assignment_loss‚0.1659083943497258 2loss/SRqqt1/assignment_loss‚1.4187709308199343 1loss/SRqqt3/detection_loss‚0.8974486679952836 % _timestamp‚1.7884659145862138e+09 _step‚134 epoch‚1 _runtime‚1967.771104043 1loss/FBt2/assignment_loss‚0.10151987070835625â¯RXÒ 1loss/SRqqt2/detection_loss‚2.3000048220157625 .loss/FBt2/detection_loss‚2.797730489419057 trainer/global_step‚1649 0loss/FBt1/assignment_loss‚0.5140430081466344 /loss/FBt1/detection_loss‚2.4390590245073493 .loss/FBt4/detection_loss‚2.244276402518153 $loss/total_loss‚49.0872190141844 .loss/FRt1/detection_loss‚2.868782849925937 -loss/FRt2/detection_loss‚2.34501641295677 0loss/FRt2/assignment_loss‚3.8175286110057387 2loss/SRqqt3/assignment_loss‚1.0733579871044538 2loss/SRqqt2/assignment_loss‚2.0663852052763105 /loss/FRt4/detection_loss‚2.8263943990071616 1loss/SRqqt4/assignment_loss‚2.399992225522345 0loss/FBt3/assignment_loss‚0.1659083943497258 2loss/SRqqt1/assignment_loss‚1.4187709308199343 1loss/SRqqt3/detection_loss‚0.8974486679952836 % _timestamp‚1.7884659145862138e+09 _step‚134 epoch‚1 _runtime‚1967.771104043 1loss/FBt2/assignment_loss‚0.10151987070835625 /loss/FRt3/assignment_loss‚1.695600921839056 /loss/FRt4/assignment_loss‚5.253882769382361 .loss/FRt3/detection_loss‚1.659816100353926 1loss/FBt4/assignment_loss‚0.36560843620100014 .loss/FRt1/assignment_loss‚3.98461183273431 0loss/SRqqt1/detection_loss‚2.314463702901717 0loss/SRqqt4/detection_loss‚2.782877641645345 /loss/FBt3/detection_loss‚0.7541383078477035†×úÙù‘‚(:† †¦çÔüáôŠ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.966' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.9915 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ã(»¨u‘‚(:ê †¦çÔ¶‚§Ž disk.sdb1.in‚0 disk.md0.in‚ 7.11328125 disk.rbd2.out‚ 0.49560546875 disk.sda.out‚213.96240234375 disk.nvme0n1.in‚ 327005.75 disk.sda1.in‚0 proc.cpu.threads‚17! disk.sdb2.out‚213.96240234375$ memory_percent‚30.85634708404541 cpu‚25.037932793243776% disk./.usageGB‚2974.7480125427246 disk.rbd0.out‚ 0.55859375! disk.nvme1n1.in‚ 199422.578125 disk.sdb1.out‚0! disk.sda2.out‚213.96240234375( disk./.usagePercent‚83.1751994594615 disk.sdb.in‚ 3.49609375 disk.sda.in‚ 3.6171875 disk.dm-0.out‚ 25403.4453125 disk.sda2.in‚ 3.6171875* proc.memory.availableMB‚11328.49609375 disk.sdb.out‚213.96240234375* proc.memory.percent‚18.582797050476074 network.recv‚688611 disk.md0.out‚ 206.6796875 disk.rbd0.in‚0.015625! disk.nvme1n1.out‚ 16412.296875! disk.nvme0n1.out‚ 8991.1484375 disk.sda1.out‚0# proc.memory.rssMB‚ 3044.60546875 network.sent‚ 2.914901e+06 disk.dm-0.in‚ 526428.328125 disk.md0p1.in‚ 7.11328125 disk.md0p1.out‚ 206.6796875 disk.sdb2.in‚ 3.49609375 disk.rbd2.in‚0q‹ÍÖ‘’‚(:† •¦çÔ¾ ¿‡ gpu.0.gpu‚43.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚116.306' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.0765 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gý2âl“‚(:á •¦çÔ¯Âʇ disk.sda.in‚ 3.6171875 disk.rbd2.out‚ 0.55859375" disk.nvme1n1.out‚ 16491.8046875 disk.dm-0.in‚527042.23828125 network.recv‚694286 disk.sdb1.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 network.sent‚ 3.021934e+06# disk.nvme1n1.in‚199553.55078125 disk.rbd0.out‚ 0.55859375$ memory_percent‚30.85794448852539 disk.rbd2.in‚0 disk.sdb.in‚ 3.49609375 disk.nvme0n1.in‚ 327488.6875 disk.sda1.out‚0 disk.sdb2.in‚ 3.49609375( proc.memory.availableMB‚ 11328.234375 cpu‚25.03782410164458( disk./.usagePercent‚83.1822180424567% disk./.usageGB‚2974.9990310668945 disk.sdb1.out‚0 disk.md0p1.in‚ 7.11328125* proc.memory.percent‚18.582797050476074 disk.sda.out‚ 215.525390625! disk.nvme0n1.out‚ 9051.2265625 disk.sda2.in‚ 3.6171875 disk.sda2.out‚ 215.525390625 disk.sdb2.out‚ 215.525390625 disk.md0.out‚ 208.1640625 disk.sdb.out‚ 215.525390625 disk.rbd0.in‚0.015625 disk.dm-0.out‚ 25543.03125 disk.md0p1.out‚ 208.1640625 disk.md0.in‚ 7.11328125# proc.memory.rssMB‚ 3044.60546875+s[Ô’”‚(:‡ ¤¦çÔØ逆 gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚117.193' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.29825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G„Ôˆ{•‚(:𠤦çÔô disk.sdb2.in‚ 3.49609375# disk.nvme1n1.out‚16624.93359375 disk.dm-0.in‚527042.23828125 disk.dm-0.out‚ 25733.546875 disk.md0.in‚ 7.11328125% memory_percent‚30.858278274536133" disk.nvme0n1.out‚ 9108.61328125* proc.memory.percent‚18.582797050476074 network.recv‚698405 disk.rbd2.out‚ 0.55859375 disk.md0p1.in‚ 7.11328125 disk.md0p1.out‚ 209.48828125 network.sent‚ 3.027768e+06 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda.out‚216.90478515625! disk.sda2.out‚216.90478515625) proc.memory.availableMB‚ 11328.1796875 proc.cpu.threads‚17 disk.sdb.in‚ 3.49609375) disk./.usagePercent‚83.18218967075515# disk.nvme1n1.in‚199553.55078125 disk.sda1.in‚0 disk.sdb.out‚216.90478515625 disk.nvme0n1.in‚ 327488.6875 disk.sda1.out‚0 disk.md0.out‚ 209.48828125 cpu‚25.03756005051483! disk.sdb2.out‚216.90478515625$ disk./.usageGB‚2974.998016357422 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 0.55859375 disk.sda2.in‚ 3.6171875# proc.memory.rssMB‚ 3044.60546875 disk.rbd2.in‚0 disk.sda.in‚ 3.6171875‰Ø*|2–- _wandbruntime‚2022  _runtime‚2022ôvÄÇ—Á & lr-AdamW/pg1‚0.0008203342354935603 & lr-AdamW/pg2‚0.0008203342354935603 trainer/global_step‚1699 % _timestamp‚1.7884659696759033e+09 _runtime‚2022.860982259 _step‚135{†Ì˜Æ & lr-AdamW/pg1‚0.0008203342354935603 & lr-AdamW/pg2‚0.0008203342354935603 trainer/global_step‚1699 % _timestamp‚1.7884659696759033e+09 _runtime‚2022.860982259 _step‚135‡lt§Ç2™- _wandbruntime‚2022  _runtime‚2022@Ì}VÇ'šÁO Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚72066266003968 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3428332253696 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚76220995213312 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3358817971781 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚68698134169600 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚68698134169600 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32049055 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚72066517674496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚78090387 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 $ _timestamp‚1.788465969677018e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318754816 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 _step‚136 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚68698152172544 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚68784289500597 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚2022.862471096 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31280886 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46067674 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3428037438976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚65425454489456 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46067674 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294814720 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3368365501952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31281577 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19108086 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3358588990705 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32049057 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3368365501952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19108084 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚76220676458496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚72066266003968 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46041332 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3368131834368 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚78116731 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚72792662959616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚68784043480161 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46041332 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3368131834368 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚78116731 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚72792662959616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚68784043480161 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚72066266003968 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3428332253696 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚76220995213312 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3358817971781 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚68698134169600 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚68698134169600 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32049055 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚72066517674496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚78090387 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 $ _timestamp‚1.788465969677018e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318754816 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 _step‚136 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚68698152172544 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚68784289500597ˆ æl 2œ- _wandbruntime‚2023  _runtime‚2023†ä¨Ò&ÌM aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3430321617408 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288509952 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚1699 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚72107767750656 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚68737438209536 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32067332 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312450048 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚68737420206592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3370095860224 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78161724 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3360776460975 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78161724 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 _step‚137 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46068024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32067332 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31298844 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚68823778347215 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3370329541120 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚72107767750656 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19118888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12179958 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚76264668547072 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3370329541120 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46094392 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3430033107456 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884659707885396e+09 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32067330 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚701 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚72834346929664 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46094392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚65463001886240 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32067330 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚703 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78135354 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚76264356097024 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚72834322989568 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚1699 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚72107767750656 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚68737438209536 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32067332 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312450048 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚68737420206592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3370095860224 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78161724 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3360776460975 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78161724 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 _step‚137 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46068024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32067332 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31298844 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚68823778347215 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3370329541120 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚72107767750656 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19118888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12179958 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚76264668547072 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3370329541120 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46094392 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3430033107456 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884659707885396e+09 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32067330 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚701 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚72834346929664 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46094392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚65463001886240 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32067330 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚703 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78135354 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚76264356097024 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚72834322989568 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚74128783560192 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚67301972360656 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47405626 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚70662436019200 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3528168588288 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚70758499391869 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32168814 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚74876837521920 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47405626 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3466347540992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚70758745412305 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32962805 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47379284 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32168130 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚74129035230720 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚1749 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚74129035230720 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚74876861462016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3466581208576t‘îÍ'¯ÇO UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚74129035230720 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚1749 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚74129035230720 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚74876861462016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3466581208576 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚682 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚74128783560192 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚67301972360656 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47405626 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚70662436019200 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3528168588288 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚70758499391869 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32168814 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚74876837521920 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47405626 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3466347540992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚70758745412305 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32962805 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47379284 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32168130 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0Œû2°- _wandbruntime‚2080  _runtime‚2080²Fe–±ÌM VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚70701722056192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3468311566848 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚74918521491968 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚70798234258923 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47405976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3530157564928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 _runtime‚2080.890453688 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32981082 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47432344 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚67339502718080 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚141 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚70701722056192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32981082 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3529873249280 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47405976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80413426 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80413426 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3458714501483 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3468545247744 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19658988 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308255744 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚78448702996992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32185468 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚70701740059136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80387056 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3468545247744 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19658986 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocateˆbù< d‚67339519757440 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32981080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32981080 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32186104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12527116 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1749 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 % _timestamp‚1.7884660277035506e+09 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104•¥xm×&²ÑM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3458714501483 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3468545247744 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19658988 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308255744 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚78448702996992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32185468 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚70701740059136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80387056 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3468545247744 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19658986 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚67339519757440 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32981080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32981080 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32186104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12527116 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1749 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 % _timestamp‚1.7884660277035506e+09 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚70701722056192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3468311566848 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚74918521491968 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚70798234258923 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47405976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3530157564928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 _runtime‚2080.890453688 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32981082 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47432344 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚67339502718080 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚141 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚70701722056192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32981082 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3529873249280 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47405976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80413426 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80413426 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368–V2³- _wandbruntime‚2080  _runtime‚2080ƬƒÅX´Ò 2loss/SRqqt4/assignment_loss‚2.1091557205827147 1loss/FBt1/assignment_loss‚0.08632308334185584 1loss/FBt2/assignment_loss‚0.18227816751875384 /loss/FBt3/detection_loss‚0.7487323599325234 _runtime‚ 2080.89331544 /loss/FRt2/assignment_loss‚4.173574205563993 0loss/FRt3/assignment_loss‚1.8133254647254944 2loss/SRqqt2/assignment_loss‚2.1048940415893282 0loss/SRqqt2/detection_loss‚2.478538475717817 0loss/SRqqt4/detection_loss‚2.724346273475223 trainer/global_step‚1749 2loss/SRqqt3/assignment_loss‚1.2273856903064169 .loss/FRt4/detection_loss‚3.078080517274362 0loss/FBt3/assignment_loss‚0.1724051819595351 /loss/FRt2/detection_loss‚2.2502643344352884 .loss/FBt2/detection_loss‚2.843611334051405 /loss/FBt4/detection_loss‚2.6432945668697356 epoch‚1 % _timestamp‚1.7884660277084184e+09 0loss/FRt1/assignment_loss‚4.0930501178458885 /loss/FRt4/assignment_loss‚4.841056536745142 2loss/SRqqt1/assignment_loss‚1.3336240503316124 0loss/SRqqt1/detection_loss‚2.009342884644866 .loss/FBt1/detection_loss‚3.076193710168203 1loss/SRqqt3/detection_loss‚1.2289680629554722 %loss/total_loss‚50.20546893139088 .loss/FRt1/detection_loss‚3.168715872146465 1loss/FBt4/assignment_loss‚0.26183515321172307 /loss/FRt3/detection_loss‚1.5564731259970632 _step‚142œ_ê]µ× 2loss/SRqqt4/assignment_loss‚2.1091557205827147 1loss/FBt1/assignment_loss‚0.08632308334185584 1loss/FBt2/assignment_loss‚0.18227816751875384 /loss/FBt3/detection_loss‚0.7487323599325234 _runtime‚ 2080.89331544 /loss/FRt2/assignment_loss‚4.173574205563993 0loss/FRt3/assignment_loss‚1.8133254647254944 2loss/SRqqt2/assignment_loss‚2.1048940415893282 0loss/SRqqt2/detection_loss‚2.478538475717817 0loss/SRqqt4/detection_loss‚2.724346273475223 trainer/global_step‚1749 2loss/SRqqt3/assignment_loss‚1.2273856903064169 .loss/FRt4/detection_loss‚3.078080517274362 0loss/FBt3/assignment_loss‚0.1724051819595351 /loss/FRt2/detection_loss‚2.2502643344352884 .loss/FBt2/detection_loss‚2.843611334051405 /loss/FBt4/detection_loss‚2.6432945668697356 epoch‚1 % _timestamp‚1.7884660277084184e+09 0loss/FRt1/assignment_loss‚4.0930501178458885 /loss/FRt4/assignment_loss‚4.841056536745142 2loss/SRqqt1/assignment_loss‚1.3336240503316124 0loss/SRqqt1/detection_loss‚2.009342884644866 .loss/FBt1/detection_loss‚3.076193710168203 1loss/SRqqt3/detection_loss‚1.2289680629554722 %loss/total_loss‚50.20546893139088 .loss/FRt1/detection_loss‚3.168715872146465 1loss/FBt4/assignment_loss‚0.26183515321172307 /loss/FRt3/detection_loss‚1.5564731259970632 _step‚142ŽLõ¹3¶‚(:„ ï¦çÔ°†–‰ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚126.22' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.555 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$#9x·‚(:í ï¦çÔÁýµ‰% memory_percent‚30.859994888305664 disk.md0.out‚ 217.3984375 disk.rbd0.in‚0.015625 network.recv‚720484 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda.out‚225.0927734375" disk.nvme0n1.out‚ 9781.33984375 disk.md0p1.in‚ 7.1171875 network.sent‚ 3.170925e+06% disk./.usageGB‚2974.9974822998047# disk.nvme0n1.in‚327585.51171875 disk.sdb2.in‚ 3.49609375 disk.sdb2.out‚225.0927734375 disk.md0.in‚ 7.1171875 cpu‚25.037115830325725 disk.dm-0.out‚27058.99609375 disk.rbd2.out‚ 0.55859375 disk.nvme1n1.out‚ 17290.28125) disk./.usagePercent‚83.18217473828065 disk.sdb.out‚225.0927734375 disk.sda.in‚ 3.62109375 disk.md0p1.out‚ 217.3984375 disk.rbd0.out‚ 0.55859375 disk.rbd2.in‚0 disk.sdb.in‚ 3.49609375 disk.sda2.in‚ 3.62109375) proc.memory.availableMB‚ 11327.8984375 disk.dm-0.in‚ 527289.515625 disk.sda2.out‚225.0927734375# proc.memory.rssMB‚ 3044.60546875 disk.sda1.out‚0* proc.memory.percent‚18.582797050476074 proc.cpu.threads‚17 disk.sda1.in‚0# disk.nvme1n1.in‚199704.00390625kj(‘¸‚(:† þ¦çÔ¼›Ý‡ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚127.446' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.8615 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î_“–u¹‚(:ê þ¦çÔâ—ˆ disk.sdb.out‚ 226.638671875 disk.dm-0.in‚ 527289.515625 disk.sdb2.out‚ 226.638671875 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.sdb1.in‚0 disk.md0p1.in‚ 7.1171875 disk.md0.in‚ 7.1171875 disk.md0.out‚ 218.88671875* proc.memory.percent‚18.582797050476074% disk./.usageGB‚2974.4672660827637 disk.sda1.in‚0 disk.sda2.in‚ 3.62109375 disk.md0p1.out‚ 218.88671875 disk.sdb1.out‚0 network.sent‚ 3.267653e+06 network.recv‚726279* proc.memory.availableMB‚11328.12109375 proc.cpu.threads‚17) disk./.usagePercent‚83.16734967093946! disk.nvme1n1.out‚ 17423.390625 disk.sdb.in‚ 3.49609375 disk.sda2.out‚ 226.638671875% memory_percent‚30.858635902404785 disk.rbd0.out‚ 0.55859375# disk.nvme0n1.in‚327585.51171875! disk.nvme0n1.out‚ 9812.3515625# proc.memory.rssMB‚ 3044.60546875 disk.sda.in‚ 3.62109375 disk.dm-0.out‚ 27235.7421875 disk.sdb2.in‚ 3.49609375 cpu‚25.03711972306648 disk.rbd2.out‚ 0.55859375# disk.nvme1n1.in‚199704.00390625 disk.sda.out‚ 226.638671875 disk.sda1.out‚0Ò +™’º‚(:‡ §çÔá ‘‡ gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.787' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚27.44675 gpu.0.smClock‚1245 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÒŽ8z»‚(:ï §çÔ½Œ’ˆ disk.rbd0.in‚0.015625 disk.nvme0n1.out‚ 9861.78125 disk.dm-0.in‚ 527289.515625* proc.memory.availableMB‚11328.12109375* proc.memory.percent‚18.582797050476074 disk.sda1.in‚0 disk.md0.out‚ 220.42578125# disk.nvme0n1.in‚327585.51171875# proc.memory.rssMB‚ 3044.60546875 disk.rbd2.out‚ 0.55859375 disk.sdb.out‚228.2236328125 disk.sdb1.in‚0 network.sent‚ 3.273461e+06$ disk./.usageGB‚2974.466972351074 disk.md0p1.in‚ 7.1171875 disk.md0.in‚ 7.1171875 disk.sda.out‚228.2236328125 disk.dm-0.out‚27418.24609375 disk.sda2.out‚228.2236328125 disk.sdb2.in‚ 3.49609375# disk.nvme1n1.in‚199704.00390625 proc.cpu.threads‚17) disk./.usagePercent‚83.16734145807848 disk.rbd0.out‚ 0.55859375# disk.nvme1n1.out‚17556.46484375 disk.sda.in‚ 3.62109375 disk.sdb.in‚ 3.49609375 disk.sdb2.out‚228.2236328125 network.recv‚730654 disk.sda1.out‚0 disk.sda2.in‚ 3.62109375 disk.md0p1.out‚ 220.42578125 disk.rbd2.in‚0% memory_percent‚30.858635902404785 disk.sdb1.out‚0 cpu‚25.037094195298042¼g`èš¼‚(: œ§çÔ•Ž‡ gpu.0.gpu‚5.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.17' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚27.292500000000004 gpu.0.smClock‚1230 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 4&$h½‚(:Ý œ§çÔŽÏ„ˆ" disk.nvme1n1.out‚ 17696.7265625 disk.sdb.out‚ 230.15625 disk.sdb1.in‚0 disk.sda1.out‚0 disk.sda2.out‚ 230.15625 network.sent‚ 3.279618e+06% disk./.usageGB‚2974.4954833984375" disk.nvme0n1.out‚ 9862.73046875 disk.sdb2.out‚ 230.15625 disk.md0.out‚ 222.29296875 disk.sda.in‚ 3.62109375 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 0.55859375 disk.rbd2.in‚0 proc.cpu.threads‚17# disk.nvme0n1.in‚327585.51171875 disk.rbd2.out‚ 0.55859375# proc.memory.rssMB‚ 3044.60546875 network.recv‚734909) disk./.usagePercent‚83.16813863889556 disk.sdb.in‚ 3.49609375 disk.sdb1.out‚0 disk.md0p1.in‚ 7.1171875% memory_percent‚30.858659744262695# disk.nvme1n1.in‚199704.00390625 disk.sda.out‚ 230.15625 disk.dm-0.in‚ 527289.515625 disk.dm-0.out‚27559.45703125 disk.sda2.in‚ 3.62109375 disk.md0.in‚ 7.1171875) proc.memory.availableMB‚ 11328.1171875 disk.sda1.in‚0 disk.md0p1.out‚ 222.29296875 cpu‚25.037070583488497 disk.sdb2.in‚ 3.49609375* proc.memory.percent‚18.582797050476074”“ü2¾- _wandbruntime‚2137  _runtime‚2137 R,XÆ¿À & lr-AdamW/pg2‚0.0008184861772741295 trainer/global_step‚1799 $ _timestamp‚1.788466084766277e+09 _runtime‚2137.951222357 _step‚143 & lr-AdamW/pg1‚0.0008184861772741295ä3kËÀÅ & lr-AdamW/pg1‚0.0008184861772741295 & lr-AdamW/pg2‚0.0008184861772741295 trainer/global_step‚1799 $ _timestamp‚1.788466084766277e+09 _runtime‚2137.951222357 _step‚143€&X:2Á- _wandbruntime‚2137  _runtime‚2137LS#Ç'ÂÁO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚80588745073664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚72733201324013 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3564796915200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚69178507271216 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3564563247616 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290620416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 $ _timestamp‚1.788466084767401e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48743578 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚650 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3564796915200 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚76191552786944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33876557 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20188284 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚72626755871744 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82620135 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33056435 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚76191301116416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚76191552786944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33876557 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3627709049344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76191301116416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12868799 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚648 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚80589059634176 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚76961036024320 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3554465071721 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314560512 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚72626737868800 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚76961059964416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33876555 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 trainer/global_step‚1799 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚72626755871744 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33057085 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20188286 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496íÍÕǪÃÆO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 $ _timestamp‚1.788466084767401e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48743578 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JBDe¼¡L"%viceStatsMonitor.on_train_batch_start/inactive_split.all.current‚650 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3564796915200 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚76191552786944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33876557 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20188284 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚72626755871744 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82620135 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33056435 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚76191301116416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚76191552786944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33876557 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3627709049344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76191301116416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12868799 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚648 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚80589059634176 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚76961036024320 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3554465071721 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314560512 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚72626737868800 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚76961059964416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33876555 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 trainer/global_step‚1799 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚72626755871744 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33057085 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20188286 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚80588745073664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚72733201324013 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3564796915200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚69178507271216 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3564563247616 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290620416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582|Ù³2Ä- _wandbruntime‚2139  _runtime‚2139VQŒéÒ&ÅÌM ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚72666041908736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚77002743934464 _step‚145 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82665128 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82638758 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚80632744403968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12875728 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚76232551179264 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82638758 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82665128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚671 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48770296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚673 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚80632429856768 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚72772690170631 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33894830 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚72666023905792 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚69216037628640 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48770296 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚76232802863104 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚1799 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _runtime‚2139.095197217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290607104 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚72772444149987 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3556423560707 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3566527273472 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33894832 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚76232802863104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33074143 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20199086 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3566527273472 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33894830 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360ˆ¹²^×&ÆÑM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82638758 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82665128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚671 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48770296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚673 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚80632429856768 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚72772690170631 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33894830 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚72666023905792 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚69216037628640 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48770296 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚76232802863104 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚1799 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _runtime‚2139.095197217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290607104 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚72772444149987 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3556423560707 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3566527273472 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33894832 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚76232802863104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33074143 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20199086 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3566527273472 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33894830 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚72666041908736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚77002743934464 _step‚145 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82665128 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82638758 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚80632744403968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12875728 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚76232551179264 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712‘®*Q2Ç- _wandbruntime‚2139  _runtime‚2139)¦vVÈÐ /loss/FRt1/assignment_loss‚3.517347821344932 /loss/FRt4/assignment_loss‚4.105860397733491 .loss/FBt4/detection_loss‚2.858254987001419 1loss/FBt1/assignment_loss‚0.32039862695253557 1loss/FBt2/assignment_loss‚0.16208205671541187 1loss/SRqqt3/detection_loss‚0.7932701525837182 0loss/SRqqt4/detection_loss‚2.227637033308706 0loss/FRt3/assignment_loss‚1.3853144600535885 1loss/SRqqt1/assignment_loss‚1.741170298035552 1loss/SRqqt2/assignment_loss‚2.079553576807181 0loss/FBt3/assignment_loss‚0.2264786122147631 /loss/FRt3/detection_loss‚1.5221125558018684 /loss/FBt1/detection_loss‚3.2340246306525335 %loss/total_loss‚46.95944622330595 trainer/global_step‚1799 1loss/FBt4/assignment_loss‚0.24001803646374356 1loss/SRqqt1/detection_loss‚2.3364574429465503 _runtime‚2139.101633956 _step‚146 .loss/FBt2/detection_loss‚2.526434553994073 /loss/FBt3/detection_loss‚0.6412331257015467 epoch‚1 2loss/SRqqt3/assignment_loss‚1.1231935732159763 2loss/SRqqt4/assignment_loss‚2.0735150099762025 .loss/FRt1/detection_loss‚2.444702014947931 .loss/FRt2/detection_loss‚2.424218712035905 0loss/SRqqt2/detection_loss‚2.298508844441838 $ _timestamp‚1.788466085915124e+09 0loss/FRt2/assignment_loss‚3.7757305150682274 .loss/FRt4/detection_loss‚2.901929185308259½Üu¬[ÉÕ .loss/FBt2/detection_loss‚2.526434553994073 /loss/FBt3/detection_loss‚0.6412331257015467 epoch‚1 2loss/SRqqt3/assignment_loss‚1.1231935732159763 2loss/SRqqt4/assignment_loss‚2.0735150099762025 .loss/FRt1/detection_loss‚2.444702014947931 .loss/FRt2/detection_loss‚2.424218712035905 0loss/SRqqt2/detection_loss‚2.298508844441838 $ _timestamp‚1.788466085915124e+09 0loss/FRt2/assignment_loss‚3.7757305150682274 .loss/FRt4/detection_loss‚2.901929185308259 /loss/FRt1/assignment_loss‚3.517347821344932 /loss/FRt4/assignment_loss‚4.105860397733491 .loss/FBt4/detection_loss‚2.858254987001419 1loss/FBt1/assignment_loss‚0.32039862695253557 1loss/FBt2/assignment_loss‚0.16208205671541187 1loss/SRqqt3/detection_loss‚0.7932701525837182 0loss/SRqqt4/detection_loss‚2.227637033308706 0loss/FRt3/assignment_loss‚1.3853144600535885 1loss/SRqqt1/assignment_loss‚1.741170298035552 1loss/SRqqt2/assignment_loss‚2.079553576807181 0loss/FBt3/assignment_loss‚0.2264786122147631 /loss/FRt3/detection_loss‚1.5221125558018684 /loss/FBt1/detection_loss‚3.2340246306525335 %loss/total_loss‚46.95944622330595 trainer/global_step‚1799 1loss/FBt4/assignment_loss‚0.24001803646374356 1loss/SRqqt1/detection_loss‚2.3364574429465503 _runtime‚2139.101633956 _step‚146’×ÒQüÊ‚(:„ «§çÔƒÁ¶ˆ gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚96.756' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚24.189 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ÷*RË‚(:Ý «§çÔµéˆ disk.md0p1.in‚ 7.12109375# disk.nvme1n1.in‚199703ìS½4.00390625% memory_percent‚30.858540534973145 proc.cpu.threads‚17 network.recv‚738552) disk./.usagePercent‚83.16819580894078 disk.sdb.in‚ 3.49609375 disk.md0.in‚ 7.12109375 disk.rbd0.out‚ 0.55859375 disk.sdb2.out‚ 231.98046875 disk.md0.out‚ 224.0546875 disk.rbd2.in‚0$ disk./.usageGB‚2974.497528076172 disk.sdb.out‚ 231.98046875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.dm-0.out‚27768.63671875# disk.nvme0n1.in‚327585.51171875 disk.sda2.out‚ 231.98046875 disk.rbd2.out‚ 0.55859375 disk.md0p1.out‚ 224.0546875* proc.memory.availableMB‚11328.13671875 network.sent‚ 3.295444e+06 disk.sda.out‚ 231.98046875! disk.nvme1n1.out‚ 17822.734375* proc.memory.percent‚18.582797050476074" disk.nvme0n1.out‚ 9945.90234375 disk.sda2.in‚3.625 disk.rbd0.in‚0.015625 disk.sda1.in‚0 disk.sdb2.in‚ 3.49609375 disk.sda.in‚3.625 disk.dm-0.in‚ 527289.515625 disk.sda1.out‚0# proc.memory.rssMB‚ 3044.60546875 cpu‚25.03702583101304ñ·ÿÌ‚(:… º§çÔÊቈ gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚142.036' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.509 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïQ% dÍ‚(:Ù º§çÔâÿÀˆ" disk.nvme1n1.out‚ 18022.5390625 disk.sdb.in‚ 3.49609375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚ 233.43359375 disk.md0.out‚ 225.44921875* proc.memory.percent‚18.582797050476074 disk.sdb2.out‚ 233.43359375 disk.dm-0.in‚527349.2578125# proc.memory.rssMB‚ 3044.60546875 disk.sda.out‚ 233.43359375" disk.nvme0n1.in‚327611.6015625! disk.nvme0n1.out‚ 10034.390625 cpu‚25.03689422725223 network.sent‚ 3.392166e+06$ disk./.usageGB‚2974.622512817383 disk.rbd2.out‚ 0.55859375 disk.nvme1n1.in‚ 199737.65625 network.recv‚744624) disk./.usagePercent‚83.17169043461521 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 0.55859375 disk.sdb.out‚ 233.43359375 disk.sda2.in‚3.625 disk.md0p1.out‚ 225.44921875 disk.md0.in‚ 7.12109375 disk.sda.in‚3.625 disk.md0p1.in‚ 7.12109375 disk.sdb2.in‚ 3.49609375% memory_percent‚30.860233306884766( proc.memory.availableMB‚ 11327.859375 disk.rbd2.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.dm-0.out‚ 28056.9296875 proc.cpu.threads‚17ÌÙîÞ΂(:… ɧçÔž¤Ö‡ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚134.908' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.727 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ß±Ï jÏ‚(:ß ɧçÔÝœ²ˆ disk.sdb.out‚234.9912109375" disk.nvme0n1.in‚327822.0078125 disk.md0p1.out‚226.9375 disk.sdb2.in‚ 3.49609375 disk.sda2.out‚234.9912109375# disk.nvme1n1.in‚199909.21484375# disk.nvme1n1.out‚18022.66796875 disk.sdb1.out‚0 disk.dm-0.in‚527731.22265625% memory_percent‚30.858731269836426 network.recv‚748438 disk.rbd2.out‚ 0.55859375 disk.sda.in‚3.625 disk.sda.out‚234.9912109375 disk.dm-0.out‚ 28125.1796875 proc.cpu.threads‚17# disk./.usageGB‚2974.74747467041 disk.sdb2.out‚234.9912109375* proc.memory.percent‚18.582797050476074 cpu‚25.036986145099 disk.sdb1.in‚0# disk.nvme0n1.out‚10102.51171875* proc.memory.availableMB‚11328.10546875# proc.memory.rssMB‚ 3044.60546875 disk.sda2.in‚3.625 disk.md0p1.in‚ 7.12109375) disk./.usagePercent‚83.17518442032646 disk.rbd2.in‚0 disk.sdb.in‚ 3.49609375 disk.sda1.in‚0 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 0.55859375 disk.sda1.out‚0 disk.md0.in‚ 7.12109375 disk.md0.out‚226.9375 network.sent‚ 3.397085e+06ÑﰒЂ(:‡ اçÔ컡‡ gpu.0.gpu‚42.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚137.693' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.42325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0WÊ@bÑ‚(:× اçÔƒâû‡ disk.sdb1.in‚0 disk.dm-0.in‚527731.22265625 disk.sda1.out‚0 disk.sdb2.out‚236.8623046875% memory_percent‚30.858802795410156 disk.md0p1.out‚228.75# disk.nvme1n1.out‚18155.75390625 disk.rbd0.in‚0.015625 disk.sda.in‚3.625 disk.sda.out‚236.8623046875# proc.memory.rssMB‚ 3044.60546875 network.recv‚752301 disk.sda1.in‚0 disk.md0.in‚ 7.12109375 disk.md0.out‚228.75 cpu‚25.0367464953714 disk.sdb1.out‚0% disk./.usageGB‚2973.9403686523438 disk.rbd2.in‚0 disk.sda2.out‚236.8623046875* proc.memory.percent‚18.582797050476074 disk.sdb.in‚ 3.49609375# disk.nvme0n1.out‚10219.87109375 proc.cpu.threads‚17 disk.md0p1.in‚ 7.12109375 network.sent‚ 3.403084e+06) disk./.usagePercent‚83.15261739826121# disk.nvme1n1.in‚199909.21484375 disk.sda2.in‚3.625" disk.nvme0n1.in‚327822.0078125 disk.sdb.out‚236.8623046875 disk.rbd0.out‚ 0.55859375 disk.rbd2.out‚ 0.55859375 disk.dm-0.out‚ 28375.625 disk.sdb2.in‚ 3.49609375' proc.memory.availableMB‚ 11328.09375HB'š2Ò- _wandbruntime‚2193  _runtime‚2193ëä$‡ÇÓÁ trainer/global_step‚1849 % _timestamp‚1.7884661401899567e+09 _runtime‚2193.374862005 _step‚147 & lr-AdamW/pg1‚0.0008174350515515247 & lr-AdamW/pg2‚0.0008174350515515247¤Ê‹rÌÔÆ & lr-AdamW/pg1‚0.0008174350515515247 & lr-AdamW/pg2‚0.0008174350515515247 trainer/global_step‚1849 % _timestamp‚1.7884661401899567e+09 _runtime‚2193.374862005 _step‚147“³ƒM€2Õ- _wandbruntime‚2193  _runtime‚2193~ü§Æ'ÖÀO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13213492 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33941876 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _step‚148 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚683 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚78253818672640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34790307 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚74591057721344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3727876383232 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚78254070343168 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚84845493 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34790307 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34790305 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50081530 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13214173 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚71055025142416 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3652403112229 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 2193.37594588 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3662778954240 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328960 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20728386 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚78253818672640 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3652632093305 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚84871837 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 trainer/global_step‚1849 $ _timestamp‚1.788466140190945e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚74707411215285 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚74591039718400 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3662778954240 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50055188 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308269056 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚74591039718400 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚79045234526720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496ÑÝ7 Ë'×ÅO WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _step‚148 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚683 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚78253818672640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34790307 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚74591057721344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3727876383232 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚78254070343168 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚84845493 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34790307 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34790305 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50081530 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13214173 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚71055025142416 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3652403112229 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 2193.37594588 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3662778954240 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328960 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20728386 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚78253818672640 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3652632093305 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚84871837 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 trainer/global_step‚1849 $ _timestamp‚1.788466140190945e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚74707411215285 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚74591039718400 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3662778954240 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50055188 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308269056 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚74591039718400 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚79045234526720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13213492 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33941876 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712”D!2Ø- _wandbruntime‚2194  _runtime‚2194/³ŽåÙÌM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚82816816752640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 _runtime‚2194.307846662 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚74630325755392 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 trainer/global_step‚1849 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33959320 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34808582 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34808582 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50108248 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84890460 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3664976660992 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3664976660992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84916830 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚74630325755392 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚79086918496768 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3654361601215 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚74630343758336 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315648 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ]JDeviceStatsMonitor.on_train_batch_˜ÀŠíend/allocated_bytes.large_pool.allocated‚74630343758336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13220822 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚74746900061695 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50081880 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20739188 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3729874315776 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3664742980096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13220134 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚71092538460480 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚78295320419328 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308255744 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚74747146082339 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚71092555499840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3654590582499 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50108248 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84890460 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3729590000128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 _step‚149 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚78295068735488 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34808580 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚690 % _timestamp‚1.7884661411222413e+09 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 trainer/global_step‚1849 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33959320 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34808582 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34808582 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50108248 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84890460 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3664976660992 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3664976660992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84916830 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚74630325755392 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚79086918496768 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3654361601215 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚74630343758336 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315648 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚74630343758336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13220822 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚74746900061695 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50081880 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20739188 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3729874315776 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3664742980096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13220134 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚71092538460480 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚78295320419328 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308255744 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚74747146082339 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚71092555499840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3654590582499 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50108248 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84890460 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3729590000128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 _step‚149 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚78295068735488 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34808580 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚690 % _timestamp‚1.7884661411222413e+09 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 _runtime‚2194.307846662•u3¸Ë2Û- _wandbruntime‚2194  _runtime‚2194ÂÉ¢\ÜÖ /loss/FRt2/assignment_loss‚4.028221349353376 2loss/SRqqt3/assignment_loss‚1.2295962013880308 1loss/SRqqt1/detection_loss‚1.8663649267476539 1loss/SRqqt2/detection_loss‚3.1388198639217175 /loss/FBt4/detection_loss‚2.3234654573293834 2loss/SRqqt1/assignment_loss‚1.8384388756492864 0loss/FRt3/assignment_loss‚1.4381318337777083 0loss/FBt1/assignment_loss‚0.3883157999745371 1loss/FBt2/assignment_loss‚0.30548928264097946 /loss/FRt4/detection_loss‚3.3730773197280035 1loss/SRqqt3/detection_loss‚0.9324889764496961 %loss/total_loss‚48.26886253686954 2loss/SRqqt2/assignment_loss‚2.3275396251364757 1loss/FBt3/assignment_loss‚0.17246186509671468 /loss/FRt2/detection_loss‚2.1755567473584327 1loss/SRqqt4/detection_loss‚2.3407551557929427 /loss/FBt2/detection_loss‚2.1276813605252434 _runtime‚2194.309405359 /loss/FBt4/assignment_loss‚0.517221705283742 .loss/FRt1/detection_loss‚2.807011839747429 0loss/FRt1/assignment_loss‚3.7379000306129457 2loss/SRqqt4/assignment_loss‚1.9238135086165533 .loss/FBt1/detection_loss‚2.450545464243208 /loss/FBt3/detection_loss‚0.7815155250235246 % _timestamp‚1.7884661411246974e+09 trainer/global_step‚1849 /loss/FRt4/assignment_loss‚4.461497320069207 epoch‚1 /loss/FRt3/detection_loss‚1.5829525024027509 _step‚150 ϸ…aÝÛ _step‚150 /loss/FRt2/assignment_loss‚4.028221349353376 2loss/SRqqt3/assignment_loss‚1.2295962013880308 1loss/SRqqt1/detection_loss‚1.8663649267476539 1loss/SRqqt2/detection_loss‚3.1388198639217175 /loss/FBt4/detection_loss‚2.3234654573293834 2loss/SRqqt1/assignment_loss‚1.8384388756492864 0loss/FRt3/assignment_loss‚1.4381318337777083 0loss/FBt1/assignment_loss‚0.3883157999745371 1loss/FBt2/assignment_loss‚0.30548928264097946 /loss/FRt4/detection_loss‚3.3730773197280035 1loss/SRqqt3/detection_loss‚0.9324889764496961 %loss/total_loss‚48.26886253686954 2loss/SRqqt2/assignment_loss‚2.3275396251364757 1loss/FBt3/assignment_loss‚0.17246186509671468 /loss/FRt2/detection_loss‚2.1755567473584327 1loss/SRqqt4/detection_loss‚2.3407551557929427 /loss/FBt2/detection_loss‚2.1276813605252434 _runtime‚2194.309405359 /loss/FBt4/assignment_loss‚0.517221705283742 .loss/FRt1/detection_loss‚2.807011839747429 0loss/FRt1/assignment_loss‚3.7379000306129457 2loss/SRqqt4/assignment_loss‚1.9238135086165533 .loss/FBt1/detection_loss‚2.450545464243208 /loss/FBt3/detection_loss‚0.7815155250235246 % _timestamp‚1.7884661411246974e+09 trainer/global_step‚1849 /loss/FRt4/assignment_loss‚4.461497320069207 epoch‚1 /loss/FRt3/detection_loss‚1.5829525024027509–ÚX£·Þ‚(:… ç§çÔà×݆ gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚142.93' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.7325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0='HYwß‚(:ì ç§çÔýü½‰ disk.md0p1.out‚ 230.1484375! disk.sdb2.out‚238.31787109375 disk.md0.out‚ 230.1484375 disk.sdb1.out‚0% memory_percent‚30.860567092895508 disk.sdb.out‚238.31787109375! disk.sda2.out‚238.31787109375 disk.rbd2.in‚0# disk.nvme1n1.out‚18267.84765625" disk.nvme0n1.in‚327822.0078125# proc.memory.rssMB‚ 3044.60546875 network.sent‚ 3.418981e+06$ disk./.usageGB‚2973.939971923828 disk.rbd2.out‚ 0.55859375 disk.sda.out‚238.31787109375 disk.dm-0.out‚ 28575.6796875 network.recv‚756095) disk./.usagePercent‚83.15260630556587 disk.sda1.in‚0 disk.sdb.in‚ 3.49609375* proc.memory.percent‚18.582797050476074 disk.dm-0.in‚527731.22265625 disk.sda1.out‚0 disk.md0p1.in‚ 7.12109375 disk.md0.in‚ 7.12109375 disk.rbd0.in‚0.015625# disk.nvme0n1.out‚10307.83203125) proc.memory.availableMB‚ 11327.8046875 disk.sdb2.in‚ 3.49609375 disk.rbd0.out‚ 0.55859375 disk.sdb1.in‚0 cpu‚25.036576270219978 disk.sda2.in‚3.625# disk.nvme1n1.in‚199909.21484375 disk.sda.in‚3.625 proc.cpu.threads‚17’ùœÀà‚(:„ ö§çÔ°ÅLJ gpu.0.gpu‚8.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚144.212' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.053 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ËJ€yá‚(:î ö§çÔ›‰£ˆ% disk./.usageGB‚2973.9399948120117 disk.rbd2.out‚ 0.55859375 disk.md0p1.in‚ 7.12109375 disk.sdb2.in‚ 3.49609375" disk.nvme0n1.in‚327822.0078125 disk.dm-0.in‚527731.22265625 disk.sda2.in‚3.625! disk.sda2.out‚239.58251953125! disk.sdb2.out‚239.58251953125 disk.md0.in‚ 7.12109375* proc.memory.percent‚18.582797050476074 network.recv‚761916 disk.rbd0.out‚ 0.55859375 disk.sda.out‚239.58251953125 proc.cpu.threads‚17% memory_percent‚30.859017372131348* proc.memory.availableMB‚11328.05859375 disk.sdb1.in‚0) disk./.usagePercent‚83.15260694552906 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.md0.out‚ 231.36328125 disk.md0p1.out‚ 231.36328125 network.sent‚ 3.515918e+06# disk.nvme1n1.out‚18268.33984375 disk.sda.in‚3.625 disk.dm-0.out‚28684.28515625 disk.sda1.in‚0 cpu‚25.0363610688386 disk.rbd0.in‚0.015625# disk.nvme1n1.in‚199909.21484375 disk.sdb.in‚ 3.49609375 disk.sdb.out‚239.58251953125" disk.nvme0n1.out‚ 10415.9453125 disk.sda1.out‚0# proc.memory.rssMB‚ 3044.60546875Û~Yžœâ‚(:‘ …¨çÔ͑χ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&Î÷-zã‚(:ï …¨çÔز†ˆ proc.cpu.threads‚17 disk.md0p1.out‚ 233.37890625 disk.sdb.out‚241.65869140625 disk.dm-0.out‚ 28773.8203125 network.sent‚ 3.521666e+06 network.recv‚765948) disk./.usagePercent‚83.14937161160822% memory_percent‚30.860590934753418* proc.memory.availableMB‚11327.80078125 disk.sdb2.in‚ 3.49609375 disk.md0.in‚ 7.12109375 disk.rbd0.out‚ 0.55859375 disk.rbd2.out‚ 0.55859375 disk.sdb1.out‚0 disk.sda.in‚3.625! disk.sda2.out‚241.65869140625* proc.memory.percent‚18.582797050476074! disk.sdb2.out‚241.65869140625 disk.rbd0.in‚0.015625" disk.nvme0n1.in‚327822.0078125 cpu‚25.03623606703798% disk./.usageGB‚2973.8242835998535 disk.rbd2.in‚0# disk.nvme1n1.in‚199909.21484375 disk.sdb.in‚ 3.49609375 disk.sda1.in‚0 disk.sda2.in‚3.625# proc.memory.rssMB‚ 3044.60546875 disk.sda1.out‚0 disk.md0p1.in‚ 7.12109375 disk.md0.out‚ 233.37890625# disk.nvme1n1.out‚18269.92578125 disk.sdb1.in‚0# disk.nvme0n1.out‚10503.89453125 disk.dm-0.in‚527731.22265625 disk.sda.out‚241.65869140625¥62ä- _wandbruntime‚2245  _runtime‚2245\†^ ÇåÁ & lr-AdamW/pg2‚0.0008162994809430812 trainer/global_step‚1899 % _timestamp‚1.7884661925885932e+09 _runtime‚2245.773501479 _step‚151 & lr-AdamW/pg1‚0.0008162994809430812AÔºÌæÆ _runtime‚2245.773501479 _step‚151 & lr-AdamW/pg1‚0.0008162994809430812 & lr-AdamW/pg2‚0.0008162994809430812 trainer/global_step‚1899 % _timestamp‚1.7884661925885932e+09—õb<|2ç- _wandbruntime‚2245  _runtime‚2245uå$È'èÂO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3750341152737 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚76555359570944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚76555359570944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 % _timestamp‚1.7884661925896285e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚80316587899392 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚72931543013616 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35704057 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51393140 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚688 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3827816354816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚80316336228864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _step‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35704057 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚80316587899392 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚84957273324032 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚81129433029120 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 _runtime‚2245.774628625 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3827519442944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚72931525974256 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚81129456969216 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3761228328448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3761228328448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3750341152737 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚76555359570944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚76555359570944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 % _timestamp‚1.7884661925896285e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚80316587899392 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚72931543013616 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35704057 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51393140 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚688 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3827816354816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚80316336228864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _step‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35704057 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚80316587899392 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚84957273324032 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚81129433029120 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 _runtime‚2245.774628625 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3827519442944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚72931525974256 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚81129456969216 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0˜ŠÐ42ê- _wandbruntime‚2246  _runtime‚2246xÝz­Ð&ëÊM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21279286 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 trainer/global_step‚1899 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34849110 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3829804250112 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13569121 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚81171116999168 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315648 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚72969073371040 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚76594645607936 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚80357586291712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308255744 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3752299641723 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3763192367616 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 2246.70901401 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87142162 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3762958686720 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51419832 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51446200 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3762958686720 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35722332 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚81171140939264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚76721601994047 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚80357837975552 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚87142162 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35722330 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51419832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚703 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3829519934464 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚85000945189376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3752528623007 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 $ _timestamp‚1.788466193523352e+09 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34848407 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚72969056331680 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚85000636933632 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35722332 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13569822 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87168532 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35722330 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚76721355973403 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚76594627604992 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87168532 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51446200 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚80357837975552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚76594645607936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚76594627604992 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582IÛÅ0Õ&ìÏM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315648 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚72969073371040 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚76594645607936 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚80357586291712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308255744 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3752299641723 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3763192367616 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 2246.70901401 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87142162 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3762958686720 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51419832 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51446200 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3762958686720 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35722332 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚81171140939264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚76721601994047 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚80357837975552 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚87142162 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35722330 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51419832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚703 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3829519934464 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚85000945189376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3752528623007 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 $ _timestamp‚1.788466193523352e+09 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34848407 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚72969056331680 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚85000636933632 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35722332 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13569822 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87168532 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35722330 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚76721355973403 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚76594627604992 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87168532 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51446200 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚80357837975552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚76594645607936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚76594627604992 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21279286 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 trainer/global_step‚1899 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34849110 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3829804250112 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13569121 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚81171116999168 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0™üKº.2í- _wandbruntime‚2246  _runtime‚2246ßÌêZîÔ .loss/FBt1/detection_loss‚3.079914410909017 /loss/FBt3/detection_loss‚0.6385244447549151 &loss/total_loss‚46.717238272138545 /loss/FRt1/detection_loss‚2.4090521491092183 0loss/SRqqt4/detection_loss‚2.348058861313444 trainer/global_step‚1899 0loss/FRt3/assignment_loss‚1.6310051720717857 2loss/SRqqt3/assignment_loss‚1.1232496485877108 .loss/FRt3/detection_loss‚1.735331157158161 1loss/FBt1/assignment_loss‚0.19605092813435476 _runtime‚2246.710610177 /loss/FRt1/assignment_loss‚3.152375825073408 .loss/FRt4/detection_loss‚3.257823863755102 1loss/SRqqt3/detection_loss‚0.9611795517470104 /loss/FBt4/detection_loss‚2.9822556495666506 epoch‚1 % _timestamp‚1.7884661935258973e+09 _step‚154 2loss/SRqqt2/assignment_loss‚2.3707758092787117 1loss/SRqqt4/assignment_loss‚2.051641175241181 1loss/FBt4/assignment_loss‚0.12614756859838466 /loss/FRt2/detection_loss‚1.9791558522830361 0loss/SRqqt1/detection_loss‚1.909136528770129 /loss/FRt2/assignment_loss‚3.865728711380678 2loss/SRqqt1/assignment_loss‚1.3817844531188408 1loss/FBt2/assignment_loss‚0.08538526394921872 1loss/FBt3/assignment_loss‚0.11455699531635204 /loss/FBt2/detection_loss‚2.6880812181366815 /loss/FRt4/assignment_loss‚4.320407514986784 0loss/SRqqt2/detection_loss‚2.309615518897772úÆy_ïÙ 0loss/FRt3/assignment_loss‚1.6310051720717857 2loss/SRqqt3/assignment_loss‚1.1232496485877108 .loss/FRt3/detection_loss‚1.735331157158161 1loss/FBt1/assignment_loss‚0.19605092813435476 _runtime‚2246.710610177 /loss/FRt1/assignment_loss‚3.152375825073408 .loss/FRt4/detection_loss‚3.257823863755102 1loss/SRqqt3/detection_loss‚0.9611795517470104 /loss/FBt4/detection_loss‚2.9822556495666506 epoch‚1 % _timestamp‚1.7884661935258973e+09 _step‚154 2loss/SRqqt2/assignment_loss‚2.3707758092787117 1loss/SRqqt4/assignment_loss‚2.051641175241181 1loss/FBt4/assignment_loss‚0.12614756859838466 /loss/FRt2/detection_loss‚1.9791558522830361 0loss/SRqqt1/detection_loss‚1.909136528770129 /loss/FRt2/assignment_loss‚3.865728711380678 2loss/SRqqt1/assignment_loss‚1.3817844531188408 1loss/FBt2/assignment_loss‚0.08538526394921872 1loss/FBt3/assignment_loss‚0.11455699531635204 /loss/FBt2/detection_loss‚2.6880812181366815 /loss/FRt4/assignment_loss‚4.320407514986784 0loss/SRqqt2/detection_loss‚2.309615518897772 .loss/FBt1/detection_loss‚3.079914410909017 /loss/FBt3/detection_loss‚0.6385244447549151 &loss/total_loss‚46.717238272138545 /loss/FRt1/detection_loss‚2.4090521491092183 0loss/SRqqt4/detection_loss‚2.348058861313444 trainer/global_step‚1899šK.áâð‚(:… ”¨çÔ©•¼ˆ gpu.0.gpu‚19.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚124.044' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.011 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ ŠÎ”ñ‚(:ë ”¨çÔÂ÷Έ disk.sdb1.in‚0# disk.nvme0n1.out‚10679.97265625 disk.sda1.out‚0$ memory_percent‚30.86066246032715) disÓª4Nâk./.usagePercent‚83.15171846329649 disk.md0p1.out‚ 234.7734375 proc.cpu.threads‚17$ disk./.usageGB‚2973.908218383789 disk.md0.in‚ 7.12109375" disk.nvme1n1.out‚ 18356.0546875# proc.memory.rssMB‚ 3044.60546875 disk.dm-0.out‚29036.02734375 network.recv‚770275 disk.md0p1.in‚ 7.12109375 disk.sdb2.in‚ 3.49609375! disk.sdb2.out‚243.09228515625! disk.sda2.out‚243.09228515625) proc.memory.availableMB‚ 11327.7890625 cpu‚25.036001723431642 disk.sdb1.out‚0 disk.sda.in‚3.625 disk.sda2.in‚3.625 network.sent‚ 3.538413e+06# disk.nvme1n1.in‚199909.21484375" disk.nvme0n1.in‚327822.0078125 disk.sda1.in‚0 disk.md0.out‚ 234.7734375 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 0.55859375 disk.rbd2.in‚0 disk.dm-0.in‚527731.22265625 disk.rbd2.out‚ 0.55859375 disk.sdb.in‚ 3.49609375 disk.sdb.out‚243.09228515625 disk.sda.out‚243.09228515625* proc.memory.percent‚18.582797050476074Z”eò‚(:„ £¨çÔƛч gpu.0.gpu‚39.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚98.924' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚24.731 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™Œýµvó‚(:ë £¨çÔïÙ°ˆ# disk.nvme1n1.out‚18356.20703125 disk.sdb1.out‚0) proc.memory.availableMB‚ 11327.9765625% disk./.usageGB‚2973.9924850463867 disk.sda.in‚3.625 disk.dm-0.in‚527731.22265625! disk.sda2.out‚244.53369140625 proc.cpu.threads‚17 disk.sda.out‚244.53369140625 disk.sda1.in‚0! disk.sdb2.out‚244.53369140625 disk.rbd0.out‚ 0.55859375# proc.memory.rssMB‚ 3044.60546875 network.recv‚774774" disk.nvme0n1.in‚327822.0078125 disk.dm-0.out‚ 29153.4296875 disk.md0p1.out‚ 236.1640625 disk.sdb2.in‚ 3.49609375 disk.sdb.out‚244.53369140625 disk.md0p1.in‚ 7.12109375 disk.md0.out‚ 236.1640625 disk.rbd2.in‚0) disk./.usagePercent‚83.15407459445106# disk.nvme0n1.out‚10797.22265625 disk.sda1.out‚0 disk.md0.in‚ 7.12109375* proc.memory.percent‚18.582797050476074 network.sent‚ 3.633891e+06 disk.sdb.in‚ 3.49609375 disk.sdb1.in‚0 disk.rbd0.in‚0.015625$ memory_percent‚30.85951805114746# disk.nvme1n1.in‚199909.21484375 disk.sda2.in‚3.625 disk.rbd2.out‚ 0.55859375 cpu‚25.03597319036576‡"—ó’ô‚(:‡ ²¨çÔã gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚134.575' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.64375 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c3¦bõ‚(:× ²¨çÔç̈ disk.sdb.out‚ 246.359375 disk.sdb1.out‚0 disk.md0.in‚ 7.12109375 disk.rbd2.out‚ 0.55859375 disk.sdb2.out‚ 246.359375 disk.sda.out‚ 246.359375 disk.sda2.in‚3.625 disk.sdb2.in‚ 3.49609375 disk.rbd2.in‚0# disk.nvme1n1.in‚199909.21484375 disk.sdb1.in‚0 disk.sda.in‚3.625# disk.nvme0n1.out‚10885.29296875# memory_percent‚30.8610200881958# proc.memory.rssMB‚ 3044.60546875% disk./.usageGB‚2973.9975929260254 disk.md0p1.in‚ 7.12109375 disk.md0p1.out‚ 237.9296875 disk.md0.out‚ 237.9296875 disk.rbd0.in‚0.015625" disk.nvme0n1.in‚327822.0078125 disk.sda1.out‚0 cpu‚25.035964301066308 disk.sda1.in‚0 disk.sda2.out‚ 246.359375* proc.memory.availableMB‚11327.73046875* proc.memory.percent‚18.582797050476074 disk.dm-0.in‚527731.22265625 disk.dm-0.out‚29327.71484375 proc.cpu.threads‚17 network.recv‚779392 disk.rbd0.out‚ 0.55859375! disk.nvme1n1.out‚ 18442.421875 network.sent‚ 3.640346e+06) disk./.usagePercent‚83.15421741290356 disk.sdb.in‚ 3.49609375²tx­Žö‚(:ƒ Á¨çÔê¯Æ‡ gpu.0.gpu‚9.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚146.38' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.595 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÕÏ×¢p÷‚(:å Á¨çÔ˜¼û‡ disk.sdb2.in‚ 3.49609375 disk.md0.in‚ 7.12109375* proc.memory.availableMB‚11327.97265625 proc.cpu.threads‚17% disk./.usageGB‚2973.9976119995117# disk.nvme1n1.in‚199909.21484375 cpu‚25.035841014371158 disk.rbd2.in‚0 disk.sda.out‚247.8447265625 disk.dm-0.in‚527731.22265625 disk.sda1.out‚0 disk.sda2.out‚247.8447265625 disk.md0p1.in‚ 7.12109375# disk.nvme1n1.out‚18448.10546875 disk.rbd0.in‚0.015625 disk.sdb2.out‚247.8447265625 disk.rbd2.out‚ 0.55859375" disk.nvme0n1.in‚327822.0078125 disk.sda1.in‚0 disk.md0p1.out‚ 239.359375* proc.memory.percent‚18.582797050476074 network.sent‚ 3.645966e+06 network.recv‚782942 disk.rbd0.out‚ 0.55859375 disk.sdb.out‚247.8447265625 disk.sdb1.in‚0 disk.dm-0.out‚ 29479.96875$ memory_percent‚30.85954189300537 disk.sdb.in‚ 3.49609375 disk.sdb1.out‚0# disk.nvme0n1.out‚11031.86328125 disk.sda2.in‚3.625# proc.memory.rssMB‚ 3044.60546875) disk./.usagePercent‚83.15421794620623 disk.md0.out‚ 239.359375 disk.sda.in‚3.625eòñv2ø- _wandbruntime‚2301  _runtime‚2301Ù¨€ÇùÁ & lr-AdamW/pg1‚0.0008150797021000117 & lr-AdamW/pg2‚0.0008150797021000117 trainer/global_step‚1949 % _timestamp‚1.7884662485848083e+09 _runtime‚2301.769730874 _step‚155ï—˜ÌúÆ & lr-AdamW/pg2‚0.0008150797021000117 trainer/global_step‚1949 % _timestamp‚1.7884662485848083e+09 _runtime‚2301.769730874 _step‚155 & lr-AdamW/pg1‚0.0008150797021000117›…h<2û- _wandbruntime‚2301  _runtime‚2301C ÆEÇ'üÁO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3859444035072 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3859210367488 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52731092 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3927319991808 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318754816 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3927614806528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35718077 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚78519643417600 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36617807 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36617807 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚87141270278144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52757434 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35718768 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89375241 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚78656569059137 _runtime‚2301.770837077 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3859210367488 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚87140951523328 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚83213655471616 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3848508174321 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82378853785088 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚78656323038701 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚78519661420544 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚83213631531520 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0ÕÇã Ì'ýÆO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36617807 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚87141270278144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52757434 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35718768 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89375241 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚78656569059137 _runtime‚2301.770837077 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3859210367488 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚87140951523328 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚83213655471616 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3848508174321 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82378853785088 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚78656323038701 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚78519661420544 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚83213631531520 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3859444035072 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3859210367488 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52731092 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3927319991808 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318754816 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3927614806528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35718077 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚78519643417600 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36617807œýcÕ2þ- _wandbruntime‚2302  _runtime‚2302óŠ2ÅÿÌM QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚89393864 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3861408074240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21819388 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52757784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚707 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚87184624155648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3861174393344 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚82420103847936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13916142 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3929605552640 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚74845591242240 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚82420355531776 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚78558929454592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 % _timestamp‚1.7884662495897226e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36636080 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35735528 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296898560 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3929308654080 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚78695811885111 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89420234 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89420234 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚705 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚78558947457536 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚78558947457536 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21819386 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚83255339441664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36636080 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚74845574202880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 "z\Ðtrainer/global_step‚1949 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3861174393344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320838656 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚83255315501568 _step‚157 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚78696057905755 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚82420355531776 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52784152 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚87184944994304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3850466663515hZˆê×&€ÑM D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3861174393344 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚82420103847936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13916142 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3929605552640 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚74845591242240 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚82420355531776 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚78558929454592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 % _timestamp‚1.7884662495897226e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36636080 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35735528 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296898560 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3929308654080 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚78695811885111 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89420234 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89420234 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚705 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚78558947457536 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚78558947457536 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21819386 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚83255339441664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36636080 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚74845574202880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1949 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3861174393344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320838656 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚83255315501568 _step‚157 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚78696057905755 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚82420355531776 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52784152 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚87184944994304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3850466663515 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚89393864 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3861408074240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21819388 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52757784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚707 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚87184624155648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768’$ø2- _wandbruntime‚2302  _runtime‚2302vŽ$&]‚× /loss/FRt2/assignment_loss‚4.342707345652026 0loss/FRt4/assignment_loss‚3.2265611622068615 /loss/FRt1/detection_loss‚2.6909669961248124 /loss/FRt2/detection_loss‚2.3676156451273167 1loss/SRqqt3/detection_loss‚0.8648384176542826 /loss/FBt1/detection_loss‚2.3838200654302324 0loss/FRt3/assignment_loss‚1.4585009567439555 2loss/SRqqt3/assignment_loss‚1.1303178032459777 1loss/FBt1/assignment_loss‚0.07850578625040937 1loss/SRqqt1/detection_loss‚2.1873377046069584 .loss/FBt4/detection_loss‚2.603100527416576 %loss/total_loss‚47.71493040524463 0loss/FBt2/assignment_loss‚0.3856709138221609 /loss/FBt2/detection_loss‚2.4655157511050882 /loss/FBt3/detection_loss‚0.8116432454224841 _runtime‚2302.776818915 _step‚158 /loss/FRt3/detection_loss‚1.6270737887360156 % _timestamp‚1.7884662495921223e+09 1loss/SRqqt4/assignment_loss‚1.917503182860938 0loss/FRt1/assignment_loss‚4.2076284170150755 2loss/SRqqt2/assignment_loss‚2.4305165079422295 .loss/FRt4/detection_loss‚3.238835354646047 1loss/SRqqt2/detection_loss‚2.6503589674830437 0loss/SRqqt4/detection_loss‚2.225569641951359 epoch‚1 1loss/FBt3/assignment_loss‚0.11502272311190323 1loss/FBt4/assignment_loss‚0.21735118103441148 trainer/global_step‚1949 2loss/SRqqt1/assignment_loss‚2.0879683196544647¦øî bƒÜ 0loss/FRt1/assignment_loss‚4.2076284170150755 2loss/SRqqt2/assignment_loss‚2.4305165079422295 .loss/FRt4/detection_loss‚3.238835354646047 1loss/SRqqt2/detection_loss‚2.6503589674830437 0loss/SRqqt4/detection_loss‚2.225569641951359 epoch‚1 1loss/FBt3/assignment_loss‚0.11502272311190323 1loss/FBt4/assignment_loss‚0.21735118103441148 trainer/global_step‚1949 2loss/SRqqt1/assignment_loss‚2.0879683196544647 /loss/FRt2/assignment_loss‚4.342707345652026 0loss/FRt4/assignment_loss‚3.2265611622068615 /loss/FRt1/detection_loss‚2.6909669961248124 /loss/FRt2/detection_loss‚2.3676156451273167 1loss/SRqqt3/detection_loss‚0.8648384176542826 /loss/FBt1/detection_loss‚2.3838200654302324 0loss/FRt3/assignment_loss‚1.4585009567439555 2loss/SRqqt3/assignment_loss‚1.1303178032459777 1loss/FBt1/assignment_loss‚0.07850578625040937 1loss/SRqqt1/detection_loss‚2.1873377046069584 .loss/FBt4/detection_loss‚2.603100527416576 %loss/total_loss‚47.71493040524463 0loss/FBt2/assignment_loss‚0.3856709138221609 /loss/FBt2/detection_loss‚2.4655157511050882 /loss/FBt3/detection_loss‚0.8116432454224841 _runtime‚2302.776818915 _step‚158 /loss/FRt3/detection_loss‚1.6270737887360156 % _timestamp‚1.7884662495921223e+09 1loss/SRqqt4/assignment_loss‚1.917503182860938ž ÙPž‘„‚(:† ШçÔѪÀˆ gpu.0.gpu‚42.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚122.486' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.6215 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[Ôõþj…‚(:ß ШçÔõ®öˆ% memory_percent‚30.860328674316406# disk.nvme1n1.in‚199909.21484375 disk.sda1.in‚0 disk.md0.out‚ 240.89453125 cpu‚25.0356967972674 disk.rbd2.out‚ 0.55859375 disk.rbd0.out‚ 0.55859375 disk.sdb.in‚ 3.49609375 disk.sdb1.in‚0 disk.sda1.out‚0 disk.sda2.out‚ 249.42578125 disk.md0.in‚ 7.12109375' proc.memory.availableMB‚ 11327.84375) disk./.usagePercent‚83.14638767656159 disk.sda.out‚ 249.42578125 network.recv‚787156" disk.nvme0n1.in‚327822.0078125 disk.dm-0.in‚527731.22265625# proc.memory.rssMB‚ 3044.60546875 disk.sdb1.out‚0 disk.sda2.in‚3.625 disk.sdb2.in‚ 3.49609375* proc.memory.percent‚18.582797050476074 proc.cpu.threads‚17 network.sent‚ 3.661791e+06 disk.rbd2.in‚0# disk.nvme1n1.out‚18514.80078125 disk.sdb.out‚ 249.42578125 disk.sda.in‚3.625" disk.nvme0n1.out‚ 11081.3828125% disk./.usageGB‚2973.7175636291504 disk.rbd0.in‚0.015625 disk.dm-0.out‚29596.18359375 disk.md0p1.in‚ 7.12109375 disk.md0p1.out‚ 240.89453125 disk.sdb2.out‚ 249.42578125Ó_Cè’†‚(:‡ ߨçԳȌ† gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚123.095' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.77375 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u}3•p‡‚(:å ߨçÔ¡Œ±‡ disk.md0.out‚ 242.33203125 disk.sdb.in‚ 3.49609375 disk.dm-0.out‚29729.91015625$ disk./.usageGB‚2973.917411804199 disk.md0.in‚ 7.12109375 disk.sdb1.out‚0 disk.sda1.out‚0& proc.memory.availableMB‚ 11327.8125* proc.memory.percent‚18.582797050476074! disk.sdb2.out‚250.92333984375" disk.nvme1n1.out‚ 18647.8828125 network.recv‚791932 disk.rbd2.in‚0 disk.sdb.out‚250.92333984375 disk.sdb1.in‚0 disk.sda.out‚250.92333984375 disk.md0p1.out‚ 242.33203125% memory_percent‚30.860519409179688 disk.rbd2.out‚ 0.55859375# disk.nvme1n1.in‚199909.21484375 disk.sda1.in‚0# proc.memory.rssMB‚ 3044.60546875 network.sent‚ 3.755387e+06( disk./.usagePercent‚83.1519755151789 disk.sdb2.in‚ 3.49609375 disk.rbd0.out‚ 0.55859375# disk.nvme0n1.out‚11082.02734375 disk.dm-0.in‚ 527732.25 disk.sda2.in‚3.625 disk.rbd0.in‚0.015625# disk.nvme0n1.in‚327823.03515625! disk.sda2.out‚250.92333984375 disk.md0p1.in‚ 7.12109375 cpu‚25.035716059797753 disk.sda.in‚3.625 proc.cpu.threads‚17õÎåÏ’ˆ‚(:‡ î¨çÔïþ† gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚134.575' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.64375 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 fËq‰‚(:æ î¨çÔ“÷…‡ disk.sda1.in‚0! disk.sdb2.out‚252.80615234375 disk.md0.in‚ 7.12109375# disk.nvme1n1.in‚199909.21484375 disk.sdb.out‚252.80615234375 disk.rbd2.in‚0 disk.sda.out‚252.80615234375# disk.nvme1n1.out‚18847.47265625 disk.dm-0.in‚ 527732.25* proc.memory.availableMB‚11327.57421875* proc.memory.percent‚18.582797050476074 proc.cpu.threads‚17 disk.sdb.in‚ 3.49609375# disk.nvme0n1.out‚11162.90234375 disk.dm-0.out‚ 30010.375! disk.sda2.out‚252.80615234375 network.sent‚ 3.761896e+06) disk./.usagePercent‚83.15421773288516$ disk./.usageGB‚2973.997604370117 disk.sdb1.out‚0 disk.sdb2.in‚ 3.49609375 disk.md0.out‚ 244.16015625 disk.rbd0.in‚0.015625 disk.sdb1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 3044.60546875 cpu‚25.035708409827315 network.recv‚796586 disk.rbd2.out‚ 0.55859375 disk.sda.in‚3.625 disk.md0p1.in‚ 7.12109375% memory_percent‚30.861973762512207 disk.rbd0.out‚ 0.55859375# disk.nvme0n1.in‚327823.03515625 disk.sda2.in‚3.625 disk.md0p1.out‚ 244.16015625¢¯‰’Š‚(:‡ ý¨çÔ׸¯† gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.357' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.83925 gpu.0.smClock‚1215 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 `m‹‚(:â ý¨çÔᶙ‡ disk.sdb.out‚254.2763671875 disk.dm-0.in‚ 527732.25 disk.dm-0.out‚30108.74609375 disk.sda1.out‚0 disk.nvme1n1.out‚ 18914.03125# disk.nvme0n1.out‚11194.71484375 disk.sda1.in‚0* proc.memory.availableMB‚11327.56640625 disk.md0.out‚ 245.5703125 disk.rbd0.out‚ 0.55859375 disk.sdb1.in‚0 disk.sda.out‚254.2763671875 disk.sda2.out‚254.2763671875% memory_percent‚30.862021446228027 disk.sda2.in‚3.625 disk.md0p1.out‚ 245.5703125) disk./.usagePercent‚83.17000445158396$ disk./.usageGB‚2974.562213897705 disk.sdb2.in‚ 3.49609375* proc.memory.percent‚18.582797050476074 network.sent‚ 3.767595e+06 disk.sdb2.out‚254.2763671875 disk.rbd2.in‚0# disk.nvme1n1.in‚199909.21484375 disk.sda.in‚3.625# proc.memory.rssMB‚ 3044.60546875 cpu‚25.035690135950986 network.recv‚800593# disk.nvme0n1.in‚327823.03515625 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 0.55859375 disk.sdb.in‚ 3.49609375 disk.sdb1.out‚0 disk.md0p1.in‚ 7.12109375 proc.cpu.threads‚17 disk.md0.in‚ 7.12109375Ì& 2Œ- _wandbruntime‚2355  _runtime‚2355ð³ÇÁ _runtime‚2355.834165033 _step‚159 & lr-AdamW/pg1‚0.0008137759692223989 & lr-AdamW/pg2‚0.0008137759692223989 trainer/global_step‚1999 % _timestamp‚1.7884663026491518e+09pÀ=ÌŽÆ & lr-AdamW/pg2‚0.0008137759692223989 trainer/global_step‚1999 % _timestamp‚1.7884663026491518e+09 _runtime‚2355.834165033 _step‚159 & lr-AdamW/pg1‚0.0008137759692223989Ÿ¸Q…C2- _wandbruntime‚2355  _runtime‚2355KŒK®ô$ÁO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37531557 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4027071104512 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚76684578756016 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37531555 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚689 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚85297830033920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14260080 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚84441623011840 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚89325226184704 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3946446214829 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14260767 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚89324901138432 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚80630778950409 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚84441623011840 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚84441371341312 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36609453 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 _step‚160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3957426074112 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54095386 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4027372210688 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 $ _timestamp‚1.788466302650292e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚80483963270144 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚76684561716656 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85297853974016 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91600599 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22348684 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚80483945267200 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54069044 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3946217233753 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚84441371341312 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54095386 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36608764 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325046272 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54069044 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3957659741696 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22348686 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚80483945267200 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37531555 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3957659741696 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3957426074112 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚80631024970845 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_MçfnÓpool.current‚ 233667584 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3946217233753 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚84441371341312 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54095386 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36608764 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325046272 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54069044 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3957659741696 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22348686 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚80483945267200 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37531555 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3957659741696 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3957426074112 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚80631024970845 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚84441371341312 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36609453 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 _step‚160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3957426074112 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54095386 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4027372210688 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 7eÖ“2’- _wandbruntime‚2356  _runtime‚2356ÝMºXÒ&“ÌM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91671936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22359486 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294801408 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚76722092074080 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36626391 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3948175722739 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14267574 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3959623780864 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14266905 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚671 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91671936 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚85339514003968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚80670267796819 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚84482873088000 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54095736 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22359488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54095736 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318741504 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚89368578020352 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚80523231304192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 _runtime‚2356.881268665 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚76722109113440 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91645566 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37549830 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3948404704023 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚89368896761856 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85339537944064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚80670513817463 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 trainer/global_step‚1999 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4029358817792 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14266905 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚671 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91671936 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚85339514003968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚80670267796819 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚84482873088000 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54095736 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22359488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54095736 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318741504 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚89368578020352 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚80523231304192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 _runtime‚2356.881268665 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚76722109113440 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91645566 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37549830 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3948404704023 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚89368896761856 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85339537944064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚80670513817463 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 trainer/global_step‚1999 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4029358817792 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22359486 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294801408 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚76722092074080 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36626391 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3948175722739 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14267574 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496¡̤¼‰2•- _wandbruntime‚2356  _runtime‚2356`£ˆU–Ï 2loss/SRqqt4/assignment_loss‚1.8987763812144598 /loss/FRt3/detection_loss‚1.4447455625315861 _runtime‚2356.882482311 .loss/FRt1/detection_loss‚2.851676942840699 .loss/FBt2/detection_loss‚2.363544908436862 /loss/FRt4/assignment_loss‚3.696695019801458 1loss/FBt4/assignment_loss‚0.08212251717285826 -loss/FRt4/detection_loss‚3.88084909816583 /loss/FRt1/assignment_loss‚3.579717478444499 /loss/FRt2/assignment_loss‚4.365629736014775 /loss/FBt3/detection_loss‚0.8181217797823122 trainer/global_step‚1999 _step‚162 1loss/SRqqt1/assignment_loss‚1.533349780055384 1loss/FBt2/assignment_loss‚0.08665355084965333 /loss/FRt2/detection_loss‚2.5968065297147924 .loss/FBt1/detection_loss‚2.465511987606684 .loss/FBt4/detection_loss‚2.730695832859386 % _timestamp‚1.7884663036976328e+09 0loss/FRt3/assignment_loss‚1.2796645964894975 2loss/SRqqt2/assignment_loss‚1.5977059885238607 2loss/SRqqt3/assignment_loss‚1.1153652017453202 1loss/FBt1/assignment_loss‚0.08518121954721873 1loss/SRqqt3/detection_loss‚0.9413942749163619 0loss/SRqqt4/detection_loss‚2.650272896885872 0loss/SRqqt1/detection_loss‚2.041400200376908 epoch‚1 0loss/FBt3/assignment_loss‚0.2210790523594264 0loss/SRqqt2/detection_loss‚2.470527085993025 %loss/total_loss‚46.79748762232873îltÇ—Ô 0loss/SRqqt2/detection_loss‚2.470527085993025 %loss/total_loss‚46.79748762232873 2loss/SRqqt4/assignment_loss‚1.8987763812144598 /loss/FRt3/detection_loss‚1.4447455625315861 _runtime‚2356.882482311 .loss/FRt1/detection_loss‚2.851676942840699 .loss/FBt2/detection_loss‚2.363544908436862 /loss/FRt4/assignment_loss‚3.696695019801458 1loss/FBt4/assignment_loss‚0.08212251717285826 -loss/FRt4/detection_loss‚3.8aarj“8084909816583 /loss/FRt1/assignment_loss‚3.579717478444499 /loss/FRt2/assignment_loss‚4.365629736014775 /loss/FBt3/detection_loss‚0.8181217797823122 trainer/global_step‚1999 _step‚162 1loss/SRqqt1/assignment_loss‚1.533349780055384 1loss/FBt2/assignment_loss‚0.08665355084965333 /loss/FRt2/detection_loss‚2.5968065297147924 .loss/FBt1/detection_loss‚2.465511987606684 .loss/FBt4/detection_loss‚2.730695832859386 % _timestamp‚1.7884663036976328e+09 0loss/FRt3/assignment_loss‚1.2796645964894975 2loss/SRqqt2/assignment_loss‚1.5977059885238607 2loss/SRqqt3/assignment_loss‚1.1153652017453202 1loss/FBt1/assignment_loss‚0.08518121954721873 1loss/SRqqt3/detection_loss‚0.9413942749163619 0loss/SRqqt4/detection_loss‚2.650272896885872 0loss/SRqqt1/detection_loss‚2.041400200376908 epoch‚1 0loss/FBt3/assignment_loss‚0.2210790523594264¢¹ÍÏû’˜‚(:‡ Œ©çÔÇ®ì… gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚144.489' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.12225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹Ê©Xk™‚(:à Œ©çÔ‚´œ‡ disk.sdb2.out‚256.1181640625) disk./.usagePercent‚83.17548328313747 disk.rbd2.in‚0 disk.rbd2.out‚ 0.55859375# disk.nvme1n1.in‚199909.21484375 disk.dm-0.out‚ 30367.5546875 disk.sda2.out‚256.1181640625 disk.rbd0.out‚ 0.55859375 disk.sdb.in‚ 3.49609375 disk.sdb.out‚256.1181640625 disk.dm-0.in‚ 527732.25 cpu‚25.035674931361196 disk.sda1.in‚0 disk.md0.out‚ 247.34765625 network.recv‚804524! disk.nvme1n1.out‚ 19113.953125 disk.sdb1.in‚0 proc.cpu.threads‚17 disk.md0p1.out‚ 247.34765625# disk.nvme0n1.in‚327823.03515625 disk.sda1.out‚0 disk.sda.in‚3.625 disk.sdb2.in‚ 3.49609375 disk.md0.in‚ 7.12109375% disk./.usageGB‚2974.7581634521484 disk.sda.out‚256.1181640625 disk.sda2.in‚3.625 disk.md0p1.in‚ 7.12109375$ memory_percent‚30.86414337158203' proc.memory.availableMB‚ 11327.21875* proc.memory.percent‚18.582797050476074 network.sent‚ 3.773641e+06 disk.sdb1.out‚0" disk.nvme0n1.out‚ 11253.6015625 disk.rbd0.in‚0.015625# proc.memory.rssMB‚ 3044.60546875 ”aMœš‚(:‘ ›©çÔ™Íׇ gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚122.818' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚30.704500000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z‹ô„h›‚(:Ý ›©çÔÐç‡ network.sent‚ 3.882221e+06) disk./.usagePercent‚83.18248906686881% disk./.usageGB‚2975.0087242126465 disk.sda1.in‚0 disk.sda2.in‚3.625 disk.sda1.out‚0 disk.sda2.out‚ 258.017578125 disk.sdb2.out‚ 258.017578125 disk.rbd2.out‚ 0.55859375 disk.dm-0.in‚ 527732.25# proc.memory.rssMB‚ 3044.60546875 disk.sdb2.in‚ 3.49609375 disk.sdb.in‚ 3.49609375 disk.sdb1.out‚0 disk.sda.in‚3.625 network.recv‚809288 disk.md0.in‚ 7.12109375 disk.md0.out‚ 249.203125 disk.rbd0.out‚ 0.55859375 disk.rbd2.in‚0 disk.sdb1.in‚0# disk.nvme0n1.out‚11370.85546875* proc.memory.percent‚18.582797050476074 disk.md0p1.out‚ 249.203125" disk.nvme1n1.out‚ 19247.0078125# disk.nvme0n1.in‚327823.03515625% memory_percent‚30.861186981201172( proc.memory.availableMB‚ 11327.703125 cpu‚25.035648448692857 disk.rbd0.in‚0.015625 disk.sdb.out‚ 258.017578125 proc.cpu.threads‚17 disk.md0p1.in‚ 7.12109375# disk.nvme1n1.in‚199909.21484375 disk.sda.out‚ 258.017578125 disk.dm-0.out‚30617.86328125Sƒç‘œ‚(:† ª©çÔùù¥ˆ gpu.0.gpu‚7.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ú7z‚(:ï ª©çÔ·øˆ disk.sdb1.out‚0 disk.sda1.out‚0 disk.sdb2.in‚ 3.49609375 disk.sda2.in‚3.625 disk.md0p1.out‚ 250.55859375! disk.sdb2.out‚259.44091796875 disk.rbd2.in‚0" disk.nvme1n1.out‚ 19380.2421875 disk.rbd2.out‚ 0.55859375) disk./.usagePercent‚83.18245898859877 disk.dm-0.in‚527732.4453125* proc.memory.availableMB‚11327.45703125 disk.sda.out‚259.44091796875 disk.md0.in‚ 7.12109375# disk.nvme1n1.in‚199909.21484375 disk.sdb.out‚259.44091796875 cpu‚25.035412763717904 proc.cpu.threads‚17 disk.sda.in‚3.625# disk.nvme0n1.in‚327823.23046875# disk.nvme0n1.out‚11488.23828125 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 0.55859375$ memory_percent‚30.86268901824951# proc.memory.rssMB‚ 3044.60546875 disk.dm-0.out‚30868.48046875! disk.sda2.out‚259.44091796875 disk.sdb1.in‚0* proc.memory.percent‚18.582797050476074 disk.sdb.in‚ 3.49609375 disk.sda1.in‚0 network.sent‚ 3.886296e+06 network.recv‚813713% disk./.usageGB‚2975.0076484680176 disk.md0p1.in‚ 7.12109375 disk.md0.out‚ 250.55859375œxA=2ž- _wandbruntime‚2412  _runtime‚24120ó•/ÆŸÀ trainer/global_step‚2049 % _timestamp‚1.7884663594966822e+09 _runtime‚ 2412.68170694 _step‚163 & lr-AdamW/pg1‚0.0008123885540062204 & lr-AdamW/pg2‚0.0008123885540062204gl Ë Å & lr-AdamW/pg2‚0.0008123885540062204 trainer/global_step‚2049 % _timestamp‚1.7884663594966822e+09 _runtime‚ 2412.68170694 _step‚163 & lr-AdamW/pg1‚0.0008123885540062204£¿¶}ô2¡- _wandbruntime‚2412  _runtime‚2412-ŒûÈ'¢ÂO Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚86503888897536 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚669 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚82448247116800 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4126857263616 trainer/global_step‚2049 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55406996 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚78561096627216 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4044384255337 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93852301 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38445305 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚91508885799936 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4127143689728 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 _step‚164 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚86503888897536 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93878645 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14609654 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22888784 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚667 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚91509196166144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310366208 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚82448265119744 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4055641780736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4044155274261 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚82448265119744 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38445307 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286426112 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚82605480882553 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55433338 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4055875448320 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚87382052476416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55406996 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37497771 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4055641780736 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38445305 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55433338 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚87382028536320 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚86504140568064 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚86504140568064 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚82448247116800 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14608987 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚82605234862117 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚78561079587856 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 I3,Í'£ÇO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚91508885799936 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4127143689728 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 _step‚164 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚86503888897536 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93878645 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14609654 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22888784 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚667 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚91509196166144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310366208 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚82448265119744 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4055641780736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4044155274261 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚82448265119744 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38445307 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286426112 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚82605480882553 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55433338 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4055875448320 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚87382052476416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55406996 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37497771 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4055641780736 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38445305 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55433338 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚87382028536320 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚86504140568064 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚86504140568064 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚82448247116800 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14608987 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚82605234862117 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚78561079587856 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚86503888897536 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚669 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚82448247116800 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4126857263616 trainer/global_step‚2049 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55406996 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚78561096627216 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4044384255337 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93852301 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38445305 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532¤ó$¶92¤- _wandbruntime‚2413  _runtime‚2413•‚·Ò<¥ÌM F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4129143625216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚86545138960384 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 _runtime‚2413.687927985 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4046342744531 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38463582 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55460056 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14615694 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38463580 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚78598609945280 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38463582 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318741504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚2049 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37515972 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚692 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4057839487488 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚91552561330176 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚86545390644224 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294801408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4057605806592 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4046113763247 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93923638 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93897268 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37515280 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93923638 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚82487533153792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14616384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚165 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚82487551156736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55433688 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22899588 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22899586 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚87423712506368 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚82487533153792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120_8×&¦ÑM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4057839487488 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚91552561330176 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚86545390644224 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294801408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4057605806592 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4046113763247 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93923638 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93897268 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37515280 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93923638 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚82487533153792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14616384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚165 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚82487551156736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55433688 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22899588 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22899586 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚87423712506368 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚82487533153792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4129143625216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚86545138960384 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 _runtime‚2413.687927985 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4046342744531 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38463582 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55460056 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14615694 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38463580 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚78598609945280 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38463582 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318741504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚2049 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37515972 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚692 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0¥‡S/s2§- _wandbruntime‚2413  _runtime‚2413y$'T¨Î 1loss/SRqqt4/assignment_loss‚1.754925073434909 0loss/FBt1/assignment_loss‚0.1548045071493915 1loss/SRqqt3/detection_loss‚1.1117458229555803 epoch‚1 2loss/SRqqt2/assignment_loss‚1.8818025387823583 0loss/SRqqt4/detection_loss‚2.728148363530636 .loss/FBt2/detection_loss‚2.472045862674713 /loss/FRt1/assignment_loss‚3.044506706040481 0loss/FBt4/assignment_loss‚0.3217660666369126 /loss/FRt3/detection_loss‚1.6459377130493522 _runtime‚2413.689602733 0loss/FRt3/assignment_loss‚1.5004778419222151 .loss/FRt4/assignment_loss‚4.35452480877147 1loss/FBt3/assignment_loss‚0.14430583110155792 .loss/FBt1/detection_loss‚2.273787980278333 -loss/FBt4/detection_loss‚2.19406216284808 2loss/SRqqt1/assignment_loss‚1.6115931937017955 1loss/FBt2/assignment_loss‚0.18549553654287695 /loss/FRt1/detection_loss‚3.1406638200940757 /loss/FRt4/detection_loss‚3.3843479997971477 %loss/total_loss‚47.56566975633477 % _timestamp‚1.7884663605049028e+09 /loss/FRt2/assignment_loss‚4.580569505691528 2loss/SRqqt3/assignment_loss‚0.8984384209145313 /loss/FBt3/detection_loss‚0.6112503387685138 trainer/global_step‚2049 -loss/FRt2/detection_loss‚2.26919198655126 0loss/SRqqt1/detection_loss‚2.363737498586242 1loss/SRqqt2/detection_loss‚2.9375401765108107 _step‚166åPY©Ó 1loss/SRqqt4/assignment_loss‚1.754925073434909 0loss/FBt1/assignment_loss‚0.1548045071493915 1loss/SRqqt3/detection_loss‚1.1117458229555803 epoch‚1 2loss/SRqqt2/assignment_loss‚1.8818025387823583 0loss/SRqqt4/detection_loss‚2.728148363530636 .loss/FBt2/detection_loss‚2.472045862674713 /loss/FRt1/assignment_loss‚3.044506706040481 0loss/FBt4/assignment_loss‚0.3217660666369126 /loss/FRt3/detection_loss‚1.6459377130493522 _runtime‚2413.689602733 0loss/FRt3/assignment_loss‚1.5004778419222151 .loss/FRt4/assignment_loss‚4.35452480877147 1loss/FBt3/assignment_loss‚0.14430583110155792 .loss/FBt1/detection_loss‚2.273787980278333 -loss/FBt4/detection_loss‚2.19406216284808 2loss/SRqqt1/assignment_loss‚1.6115931937017955 1loss/FBt2/assignment_loss‚0.18549553654287695 /loss/FRt1/detection_loss‚3.1406638200940757 /loss/FRt4/detection_loss‚3.3843479997971477 %loss/total_loss‚47.56566975633477 % _timestamp‚1.7884663605049028e+09 /loss/FRt2/assignment_loss‚4.580569505691528 2loss/SRqqt3/assignment_loss‚0.8984384209145313 /loss/FBt3/detection_loss‚0.6112503387685138 trainer/global_step‚2049 -loss/FRt2/detection_loss‚2.26919198655126 0loss/SRqqt1/detection_loss‚2.363737498586242 1loss/SRqqt2/detection_loss‚2.9375401765108107 _step‚166¦Ø–RÂ’ª‚(:‡ ¹©çÔ†•Ž‡ gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚133.349' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.33725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÀPÞšq«‚(:æ ¹©çÔÂÿ¼ˆ$ disk./.usageGB‚2975.008213043213 disk.sda.in‚3.625 disk.dm-0.out‚ 31145.6015625! disk.sdb2.out‚260.86376953125% memory_percent‚30.862903594970703 disk.sdb1.in‚0# disk.nvme0n1.out‚11632.31640625 disk.sdb2.in‚ 3.49609375 network.sent‚ 3.904945e+06( disk./.usagePercent‚83.1824747743575 disk.dm-0.in‚527732.4921875 disk.md0p1.in‚ 7.12109375( proc.memory.availableMB‚ 11327.421875* proc.memory.percent‚18.582797050476074 network.recv‚818028 disk.md0.out‚ 251.92578125 disk.rbd2.out‚ 0.55859375 disk.nvme1n1.in‚ 199909.21875 disk.sdb.in‚ 3.49609375 disk.sdb1.out‚0" disk.nvme0n1.in‚327823.2734375 disk.sda1.in‚0 disk.rbd0.out‚ 0.55859375 disk.sda2.in‚3.625 proc.cpu.threads‚17 disk.sdb.out‚260.86376953125 disk.sda1.out‚0 cpu‚25.0354009662947 disk.md0.in‚ 7.12109375 disk.rbd0.in‚0.015625 disk.rbd2.in‚0# disk.nvme1n1.out‚19513.28515625 disk.sda.out‚260.86376953125! disk.sda2.out‚260.86376953125 disk.md0p1.out‚ 251.92578125# proc.memory.rssMB‚ 3044.60546875é¾ýÊ’¬‚(:‡ È©çÔ‹¹†† gpu.0.gpu‚14.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚141.087' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.27175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|¹¢¬q­‚(:æ È©çÔÔô¥‡ network.sent‚ 4.001683e+06) disk./.usagePercent‚83.18247264114686 disk.rbd2.out‚ 0.55859375# disk.nvme0n1.out‚11693.55859375* proc.memory.percent‚18.582797050476074 disk.nvme1n1.in‚ 199909.21875 disk.sda.in‚3.625# disk.nvme0n1.in‚327823.27734375 disk.sda1.in‚0 disk.rbd0.in‚0.015625# proc.memory.rssMB‚ 3044.60546875% disk./.usageGB‚2975.0081367492676 disk.sda2.out‚ 262.958984375 disk.sdb2.out‚ 262.958984375 disk.sda.out‚ 262.958984375" disk.nvme1n1.out‚ 19645.5546875 disk.sdb.in‚ 3.49609375 disk.sdb1.out‚0 disk.dm-0.in‚527732.49609375* proc.memory.availableMB‚11327.42578125 disk.sda1.out‚0 disk.sda2.in‚3.625 disk.md0.out‚ 253.95703125 disk.rbd0.out‚ 0.55859375 disk.md0p1.in‚ 7.12109375 disk.md0p1.out‚ 253.95703125 cpu‚25.035311954875596 disk.rbd2.in‚0 disk.md0.in‚ 7.12109375 network.recv‚824260 disk.sdb.out‚ 262.958984375 disk.sdb1.in‚0 disk.dm-0.out‚31339.11328125 disk.sdb2.in‚ 3.49609375% memory_percent‚30.862879753112793 proc.cpu.threads‚17•íÙ’’®‚(:‡ שçÔÛ®Á‡ gpu.0.gpu‚12.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚137.693' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.42325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆm¯÷n¯‚(:ã שçÔ˨ه cpu‚25.03529142133477 disk.nvme1n1.in‚ 199909.21875 network.recv‚828233$ disk./.usageGB‚2974.477954864502 disk.rbd0.in‚0.015625* proc.memory.percent‚18.582797050476074 disk.sdb.in‚ 3.49609375 disk.sdb.out‚ 264.619140625 disk.sda.in‚3.625 disk.sda.out‚ 264.619140625 disk.dm-0.in‚527732.49609375* proc.memory.availableMB‚11327.17578125) disk./.usagePercent‚83.16764853375047 disk.rbd0.out‚ 0.55859375 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb2.out‚ 264.619140625 disk.sdb1.in‚0# disk.nvme1n1.out‚19779.53515625 disk.md0.in‚ 7.12109375 disk.dm-0.out‚31549.20703125% memory_percent‚30.864405632019043 disk.sdb2.in‚ 3.49609375 disk.sdb1.out‚0! disk.nvme0n1.out‚ 11769.671875 disk.sda2.in‚3.625 disk.sda2.out‚ 264.619140625 disk.md0.out‚ 255.55859375 proc.cpu.threads‚17 network.sent‚ 4.007008e+06 disk.rbd2.out‚ 0.55859375# disk.nvme0n1.in‚327823.27734375 disk.md0p1.in‚ 7.12109375 disk.md0p1.out‚ 255.55859375# proc.memory.rssMB‚ 3044.60546875Äö`Ç°‚(:… æ©çÔ•¢Ç‡ gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚143.872' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.968 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸcHp±‚(:å æ©çÔ þó‡ disk.sdb.in‚ 3.49609375* proc.memory.percent‚18.582797050476074 cpu‚25.035275287510117 disk.sda1.in‚0 disk.md0p1.out‚ 256.9765625 disk.sdb2.out‚266.0927734375 disk.rbd0.in‚0.015625% memory_percent‚30.862927436828613 disk.dm-0.out‚31732.94921875 disk.rbd2.in‚0 disk.nvme1n1.in‚ 199909.21875 disk.sdb.out‚266.0927734375 disk.md0p1.in‚ 7.12109375 disk.sda.in‚3.625 disk.sda.out‚266.0927734375 disk.dm-0.in‚527732.49609375 disk.sdb1.in‚0# disk.nvme0n1.in‚327823.27734375 disk.sdb2.in‚ 3.49609375 disk.md0.in‚ 7.12109375 disk.nvme1n1.out‚ 19912.9375# proc.memory.rssMB‚ 3044.60546875 disk.sda1.out‚0 disk.sda2.out‚266.0927734375# disk.nvme0n1.out‚11820.01171875 disk.md0.out‚ 256.9765625 network.recv‚832659 disk.sda2.in‚3.625 disk.rbd0.out‚ 0.55859375 disk.sdb1.out‚0* proc.memory.availableMB‚11327.41796875 proc.cpu.threads‚17 network.sent‚ 4.013285e+06) disk./.usagePercent‚83.16622568225196% disk./.usageGB‚2974.4270668029785 disk.rbd2.out‚ 0.55859375V¨ü¢2²- _wandbruntime‚2470  _runtime‚2470ÅId±Æ³À & lr-AdamW/pg1‚0.0008109177455867279 & lr-AdamW/pg2‚0.0008109177455867279 trainer/global_step‚2099 $ _timestamp‚1.788466416945436e+09 _runtime‚2470.130604827 _step‚167S+BÅË´Å & lr-AdamW/pg1‚0.0008109177455867279 & lr-AdamW/pg2‚0.0008109177455867279 trainer/global_step‚2099 $ _timestamp‚1.788466416945436e+09 _runtime‚2470.130604827 _step‚167§­i–¸2µ- _wandbruntime‚2470  _runtime‚2470Й;%¶ÂO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4226628652032 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚168 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚88566406453760 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96130347 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38386240 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56744948 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4226929758208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚84412566969344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39359055 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56771290 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39359057 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚84412548966400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚93692855690752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4142093314769 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 trainer/global_step‚2099 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚88566406453760 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56771290 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39359057 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚84412548966400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚93692855690752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4142093314769 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 trainer/global_step‚2099 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚88566406453760 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚88566406453760 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96130347 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38386240 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56744948 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4226929758208¨ÿ–ºÔ2¸- _wandbruntime‚2471  _runtime‚2471¸6¹ÜÒ&¹ÌM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39377330 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚2099 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4144051803755 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96148970 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39377330 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93736850474496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚89507911008768 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39377332 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4155821513216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56798008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14965453 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚84451853006336 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚84451835003392 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚84619179620235 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14964787 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4228633307136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚668 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4156055194112 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4155821513216 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96175340 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96148970 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282218496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39377332 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚88607656516608 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚666 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56771640 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚84451835003392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4156055194112 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38404473 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚89507934948864 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23439688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4228915525632 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96175340 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚84451853006336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56771640 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚93736544315904 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚80475127816480 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚88607908200448 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96175340 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚84451853006336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56771640 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚93736544315904 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚80475127816480 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚88607908200448 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4144051803755 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96148970 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39377330 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93736850474496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚89507911008768 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39377332 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4155821513216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56798008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14965453 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚84451853006336 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚84451835003392 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚84619179620235 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14964787 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4228633307136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚668 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4156055194112 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4155821513216 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96175340 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96148970 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282218496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39377332 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚88607656516608 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚666 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56771640 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚84451835003392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4156055194112 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38404473 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚89507934948864 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23439688©‹á#ž2»- _wandbruntime‚2471  _runtime‚2471¦ )ôZ¼Ô 2loss/SRqqt4/assignment_loss‚1.9065363786437295 1loss/FBt4/assignment_loss‚0.16176669439298302 0loss/SRqqt2/detection_loss‚2.712100989818573 1loss/SRqqt3/detection_loss‚1.0017734907807843 1loss/SRqqt4/detection_loss‚2.4762350158257918 %loss/total_loss‚47.42673468937285 trainer/global_step‚2099 % _timestamp‚1.7884664181177318e+09 /loss/FRt2/assignment_loss‚4.001200552420182 /loss/FRt3/assignment_loss‚1.470836294663919 _runtime‚2471.302674347 /loss/FRt4/assignment_loss‚4.432899212837219 _step‚170 1loss/FBt1/assignment_loss‚0.12603670394087707 1loss/FBt2/assignment_loss‚0.11759530280858144 1loss/FBt3/assignment_loss‚0.04229363268077653 .loss/FRt2/detection_loss‚2.034233295066621 .loss/FBt4/detection_loss‚2.223418030142784 0loss/FRt1/assignment_loss‚3.2787615426125063 /loss/FRt1/detection_loss‚2.8515555166429087 .loss/FBt2/detection_loss‚2.838153968254725 2loss/SRqqt3/assignment_loss‚1.3380672380288121 /loss/FRt3/detection_loss‚1.5273215482766564 /loss/FRt4/detection_loss‚3.3198243498802187 .loss/FBt1/detection_loss‚2.646950307819578 epoch‚1 2loss/SRqqt1/assignment_loss‚1.6746357596733354 1loss/SRqqt2/assignment_loss‚2.483047001361847 1loss/SRqqt1/detection_loss‚1.9686731452291661 /loss/FBt3/detection_loss‚0.7928187175702694þ˽٠_runtime‚2471.302674347 /loss/FRt4/assignment_loss‚4.432899212837219 _step‚170 1loss/FBt1/assignment_loss‚0.12603670394087707 1loss/FBt2/assignment_loss‚0.11759530280858144 1loss/FBt3/assignment_loss‚0.04229363268077653 .loss/FRt2/detection_loss‚2.034233295066621 .loss/FBt4/detection_loss‚2.223418030142784 0loss/FRt1/assignment_loss‚3.2787615426125063 /loss/FRt1/detection_loss‚2.8515555166429087 .loss/FBt2/detection_loss‚2.838153968254725 2loss/SRqqt3/assignmeA>˜Xnt_loss‚1.3380672380288121 /loss/FRt3/detection_loss‚1.5273215482766564 /loss/FRt4/detection_loss‚3.3198243498802187 .loss/FBt1/detection_loss‚2.646950307819578 epoch‚1 2loss/SRqqt1/assignment_loss‚1.6746357596733354 1loss/SRqqt2/assignment_loss‚2.483047001361847 1loss/SRqqt1/detection_loss‚1.9686731452291661 /loss/FBt3/detection_loss‚0.7928187175702694 2loss/SRqqt4/assignment_loss‚1.9065363786437295 1loss/FBt4/assignment_loss‚0.16176669439298302 0loss/SRqqt2/detection_loss‚2.712100989818573 1loss/SRqqt3/detection_loss‚1.0017734907807843 1loss/SRqqt4/detection_loss‚2.4762350158257918 %loss/total_loss‚47.42673468937285 trainer/global_step‚2099 % _timestamp‚1.7884664181177318e+09 /loss/FRt2/assignment_loss‚4.001200552420182 /loss/FRt3/assignment_loss‚1.470836294663919ª$SxT¾‚(:… õ©çÔÅáäŠ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚124.044' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.011 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×Íý÷f¿‚(:Û õ©çÔðÖö disk.md0.out‚ 258.55078125 disk.rbd2.in‚0 disk.rbd2.out‚ 0.55859375 disk.dm-0.out‚ 31946.78125 disk.sdb2.out‚267.7314453125 disk.nvme1n1.out‚20046 disk.dm-0.in‚527732.49609375* proc.memory.percent‚18.582797050476074 disk.nvme1n1.in‚ 199909.21875 disk.sdb.in‚ 3.49609375 disk.sdb.out‚267.7314453125 disk.sda.in‚3.625% memory_percent‚30.866384506225586) proc.memory.availableMB‚ 11326.8515625 proc.cpu.threads‚17 network.sent‚ 4.016928e+06 network.recv‚837091 disk.rbd0.in‚0.015625 disk.sda.out‚267.7314453125 disk.sda2.out‚267.7314453125 disk.md0p1.out‚ 258.55078125# proc.memory.rssMB‚ 3044.60546875) disk./.usagePercent‚83.16842736895602 disk.sdb2.in‚ 3.49609375 disk.sdb1.out‚0 disk.sda1.out‚0 disk.md0p1.in‚ 7.12109375 cpu‚25.035075015249102 disk.sdb1.in‚0 disk.sda1.in‚0 disk.sda2.in‚3.625 disk.md0.in‚ 7.12109375# disk.nvme0n1.in‚327823.27734375 disk.nvme0n1.out‚ 11900.78125 disk.rbd0.out‚ 0.55859375% disk./.usageGB‚2974.5058097839355“î’íœÀ‚(:‘ „ªçÔÙ£†‡ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚119.368' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.841999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mv³2pÁ‚(:å „ªçÔŸÒ‡ disk.md0p1.in‚ 7.12109375 network.sent‚ 4.1253e+06 network.recv‚841550# disk.nvme1n1.out‚20179.16015625 disk.sda.in‚3.625 disk.sda1.out‚0 disk.sdb2.in‚ 3.49609375* proc.memory.availableMB‚11327.58203125) disk./.usagePercent‚83.16909677045561" disk.nvme1n1.in‚199934.0703125# disk.nvme0n1.in‚327930.71484375# disk.nvme0n1.out‚11932.73046875 disk.sda2.in‚3.625 disk.rbd0.out‚ 0.55859375% memory_percent‚30.861926078796387 disk.rbd2.out‚ 0.55859375 disk.md0.in‚ 7.12109375 disk.sdb.in‚ 3.49609375 disk.sda2.out‚ 269.279296875 disk.sdb2.out‚ 269.279296875 disk.md0.out‚ 260.04296875* proc.memory.percent‚18.582797050476074 cpu‚25.035059463608146 proc.cpu.threads‚17 disk.sda.out‚ 269.279296875 disk.dm-0.in‚527864.78515625 disk.rbd2.in‚0 disk.sdb.out‚ 269.279296875 disk.sdb1.out‚0 disk.sda1.in‚0 disk.md0p1.out‚ 260.04296875 disk.rbd0.in‚0.015625# proc.memory.rssMB‚ 3044.60546875% disk./.usageGB‚2974.5297508239746 disk.sdb1.in‚0 disk.dm-0.out‚ 32111.890625T_Eñ’‚(:‡ “ªçÔ‘濇 gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚119.091' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.77275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÌŒõƒnÂ(:ã “ªçÔß؇! disk.sdb2.out‚270.95068359375 disk.sdb.in‚ 3.49609375 network.recv‚845391 disk.dm-0.out‚32170.74609375 disk.md0.in‚ 7.12109375" disk.nvme1n1.in‚199965.9921875) disk./.usagePercent‚83.17522751118143 disk.nvme0n1.in‚ 328074.71875 disk.md0p1.in‚ 7.12109375# proc.memory.rssMB‚ 3044.60546875* proc.memory.percent‚18.582797050476074 proc.cpu.threads‚17! disk.sda2.out‚270.95068359375 disk.rbd0.in‚0.015625 disk.sda.in‚3.625% disk./.usageGB‚2974.7490158081055 disk.md0p1.out‚ 261.65625# disk.nvme1n1.out‚20179.29296875 disk.sdb.out‚270.95068359375 network.sent‚ 4.130609e+06! disk.nvme0n1.out‚ 11991.453125 disk.rbd2.in‚0 disk.sdb1.in‚0 disk.sda.out‚270.95068359375% memory_percent‚30.861949920654297( proc.memory.availableMB‚ 11327.578125 disk.dm-0.in‚528040.7109375 disk.sda1.in‚0 disk.sdb2.in‚ 3.49609375 disk.md0.out‚ 261.65625 disk.rbd0.out‚ 0.55859375 disk.rbd2.out‚ 0.55859375 disk.sdb1.out‚0 cpu‚25.035048378232304 disk.sda1.out‚0 disk.sda2.in‚3.625ËmŠÊÄ‚(:„ ¢ªçÔ¸ÿ•‡ gpu.0.gpu‚7.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚120.31' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.0775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ɵoÅ‚(:ä ¢ªçÔö‘ø‡ disk.rbd2.out‚ 0.55859375# disk.nvme1n1.out‚20379.16796875 proc.cpu.threads‚17 disk.sda.in‚3.625 disk.sdb1.out‚0* proc.memory.percent‚18.582797050476074 network.recv‚849882) disk./.usagePercent‚83.17241113983076 disk.nvme0n1.in‚ 328074.71875 disk.nvme0n1.out‚ 12109.125 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 cpu‚25.035033150508472 disk.md0p1.in‚ 7.12109375 disk.md0p1.out‚ 263.12109375 disk.sdb2.in‚ 3.49609375 disk.md0.in‚ 7.12109375 disk.sdb1.in‚0 network.sent‚ 4.136987e+06% disk./.usageGB‚2974.6482887268066 disk.dm-0.out‚32488.29296875 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0.out‚ 263.12109375 disk.rbd0.out‚ 0.55859375 disk.sda.out‚272.4716796875 disk.dm-0.in‚528040.7109375 disk.sda2.out‚272.4716796875" disk.nvme1n1.in‚199965.9921875 disk.sdb.out‚272.4716796875% memory_percent‚30.863499641418457* proc.memory.availableMB‚11327.32421875# proc.memory.rssMB‚ 3044.60546875 disk.sda2.in‚3.625 disk.sdb2.out‚272.4716796875 disk.sdb.in‚ 3.49609375?$lú2Æ- _wandbruntime‚2528  _runtime‚2528ã2´ÌÇÇÁ _step‚171 & lr-AdamW/pg1‚0.0008093638504781912 & lr-AdamW/pg2‚0.0008093638504781912 trainer/global_step‚2149 % _timestamp‚1.7884664748660345e+09 _runtime‚2528.051120312®¿¶ÌÈÆ & lr-AdamW/pg1‚0.0008093638504781912 & lr-AdamW/pg2‚0.0008093638504781912 trainer/global_step‚2149 % _timestamp‚1.7884664748660345e+09 _runtime‚2528.051120312 _step‚171«ÚˆáA2É- _wandbruntime‚2528  _runtime‚2528Tmß¾Æ'ÊÀO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4326358637568 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40272805 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4252073193984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4326649257984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚86376850816000 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚95876784178688 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98355705 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40272807 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314560512 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290620416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 trainer/global_step‚2149 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚90628924009984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚674 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40272805 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98382049 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23968984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4252306861568 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4240260336353 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚86376850816000 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4252306861568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚90629175680512 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40272807 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚91550425541120 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚82314115330256 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚86554392705969 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚86554146685533 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39275001 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15306017 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58082900 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39275675 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4240031355277 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58082900 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23968986 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚90629175680512 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚82314132369616 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4252073193984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4326649257984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚86376850816000 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚95876784178688 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98355705 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40272807 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314560512 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290620416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 trainer/global_step‚2149 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚90628924009984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚674 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40272805 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98382049 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23968984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4252306861568 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4240260336353 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚86376850816000 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4252306861568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚90629175680512 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40272807 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚91550425541120 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚82314115330256 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚86554392705969 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚86554146685533 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39275001 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15306017 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58082900 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39275675 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4240031355277 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58082900 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23968986 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608¬–*Œ2Ì- _wandbruntime‚2529  _runtime‚2529ÒO€çwÍËM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚91592133451264 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 trainer/global_step‚2149 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15313359 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4254270900736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98427042 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39293868 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4242218825547 _step‚173 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚723 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58109592 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚90670174072832 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40291082 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚721 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚86416154855936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4241989844263 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4254037219840 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40291080 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23979788 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚82351662727040 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4328351193088 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39293145 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98427042 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚2529.218758678 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98400672 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40291082 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58135960 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4254037219840 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚86593635531943 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚82351645687680 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896﹜Ö&ÎÐM JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98427042 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚2529.218758678 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98400672 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40291082 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58135960 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4254037219840 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚86593635531943 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚82351645687680 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚91592133451264 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 trainer/global_step‚2149 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15313359 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4254270900736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98427042 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39293868 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4242218825547 _step‚173 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚723 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58109592 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚90670174072832 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40291082 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚721 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚86416154855936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4241989844263 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4254037219840 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40291080 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23979788 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚82351662727040 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4328351193088 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39293145 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 FjK‘Ò‚(:† ±ªçÔÓ꼈 gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚140.754' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.1885 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u­³3iÓ‚(:Þ ±ªçÔÖí¨‰ network.recv‚854222 disk.sdb.in‚ 3.49609375 disk.sda.in‚3.625 disk.md0p1.in‚ 7.12109375# memory_percent‚30.8621883392334) proc.memory.availableMB‚ 11327.5390625 disk.rbd0.out‚ 0.55859375 disk.rbd2.out‚ 0.55859375 disk.md0p1.out‚ 265.0546875! disk.sdb2.out‚274.47412109375" disk.nvme1n1.in‚199965.9921875 network.sent‚ 4.153473e+06 disk.sdb1.in‚0 disk.nvme0n1.out‚ 12226.375 disk.sda1.out‚0 disk.md0.out‚ 265.0546875! disk.nvme1n1.out‚ 20578.734375 disk.sda.out‚274.47412109375 proc.cpu.threads‚17) disk./.usagePercent‚83.17239546073255 disk.nvme0n1.in‚ 328074.71875 disk.dm-0.out‚ 32805.109375 disk.sda1.in‚0 disk.sda2.in‚3.625! disk.sda2.out‚274.47412109375 disk.sdb2.in‚ 3.49609375 disk.rbd2.in‚0 disk.sdb.out‚274.47412109375 disk.md0.in‚ 7.12109375# proc.memory.rssMB‚ 3044.60546875* proc.memory.percent‚18.582797050476074 cpu‚25.03489776274863% disk./.usageGB‚2974.6477279663086 disk.rbd0.in‚0.015625 disk.sdb1.out‚0 disk.dm-0.in‚528040.7109375C'ÏhÔ‚(:Ý ÀªçÔã“»‡" disk.nvme1n1.in‚199965.9921875 disk.nvme0n1.in‚ 328074.71875# disk.nvme0n1.out‚12354.56640625 network.sent‚ 4.249859e+06 disk.rbd2.in‚0! disk.nvme1n1.out‚ 20711.828125 disk.sdb.out‚275.8798828125 disk.sda.out‚275.8798828125) disk./.usagePercent‚83.17239546073255 disk.md0p1.out‚ 266.40625 disk.rbd2.out‚ 0.55859375 disk.sdb1.out‚0 disk.sda.in‚3.625% disk./.usageGB‚2974.6477279663086 disk.sdb2.in‚ 3.49609375 disk.md0.out‚ 266.40625 disk.rbd0.out‚ 0.55859375 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚18.582797050476074 network.recv‚859148 disk.sdb2.out‚275.8798828125 disk.md0.in‚ 7.12109375 disk.sdb.in‚ 3.49609375 disk.dm-0.out‚33066.39453125 disk.sda2.in‚3.625 disk.sda2.out‚275.8798828125# proc.memory.rssMB‚ 3044.60546875# memory_percent‚30.8621883392334 cpu‚25.03502316130514 disk.sdb1.in‚0 disk.dm-0.in‚528040.7109375) proc.memory.availableMB‚ 11327.5390625 proc.cpu.threads‚17 disk.md0p1.in‚ 7.12109375 disk.rbd0.in‚0.015625X†ÃΑÕ‚(:† ÀªçÔøÁȇ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚138.642' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.6605 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÑPM’Ö‚(:‡ ϪçÔ€Öû… gpu.0.gpu‚23.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚143.263' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.81575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öwŒŽsׂ(:è ϪçÔÿ ¬‡ disk.sdb.out‚277.55615234375 disk.sda1.in‚0 disk.md0p1.out‚ 268.0234375* proc.memory.percent‚18.582797050476074" disk.nvme0n1.out‚ 12402.3984375 disk.sda2.in‚3.625 network.recv‚862758% disk./.usageGB‚2973.9414443969727 disk.rbd0.out‚ 0.55859375# proc.memory.rssMB‚ 3044.60546875" disk.nvme1n1.out‚ 20844.9296875 disk.sdb1.out‚0# disk.nvme0n1.in‚328074.76953125 disk.md0p1.in‚ 7.12109375( proc.memory.availableMB‚ 11327.296875) disk./.usagePercent‚83.15264747653127 disk.sdb.in‚ 3.49609375 disk.dm-0.out‚ 33247.328125 disk.rbd2.in‚0 disk.sdb1.in‚0 disk.sda.out‚277.55615234375 disk.sda1.out‚0 disk.sdb2.in‚ 3.49609375 disk.rbd2.out‚ 0.55859375 cpu‚25.034919449759066 proc.cpu.threads‚17 network.sent‚ 4.25525e+06 disk.dm-0.in‚528040.76171875! disk.sdb2.out‚277.55615234375" disk.nvme1n1.in‚199965.9921875 disk.sda.in‚3.625! disk.sda2.out‚277.55615234375 disk.md0.in‚ 7.12109375 disk.md0.out‚ 268.0234375 disk.rbd0.in‚0.015625% memory_percent‚30.863666534423828f‚tÕØ‚(:… ÞªçÔïÖɆ gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚143.872' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.968 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³óñÆxÙ‚(:í ÞªçÔý”‡ disk.rbd0.out‚ 0.55859375# disk.nvme0n1.out‚12549.06640625 disk.dm-0.out‚ 33505.9765625 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.availableMB‚11327.29296875 network.sent‚ 4.261532e+06 disk.md0p1.out‚ 269.8984375 disk.rbd2.out‚ 0.55859375 disk.sdb1.in‚0 disk.sdb1.out‚0# disk.nvme0n1.in‚328074.76953125 disk.dm-0.in‚528040.76171875$ memory_percent‚30.86369037628174 cpu‚25.034818376562722 network.recv‚867207 disk.sdb2.in‚ 3.49609375 disk.md0.in‚ 7.12109375 disk.md0.out‚ 269.8984375 disk.sdb.out‚279.49951171875% disk./.usageGB‚2973.9409141540527 disk.rbd0.in‚0.015625 disk.sdb.in‚ 3.49609375 disk.sda.out‚279.49951171875# proc.memory.rssMB‚ 3044.60546875* proc.memory.percent‚18.582797050476074 disk.rbd2.in‚0# disk.nvme1n1.out‚20956.91015625 disk.sda2.in‚3.625) disk./.usagePercent‚83.15263265071731! disk.sdb2.out‚279.49951171875" disk.nvme1n1.in‚199965.9921875 disk.sda.in‚3.625! disk.sda2.out‚279.49951171875 proc.cpu.threads‚17 disk.md0p1.in‚ 7.12109375–~)2Ú- _wandbruntime‚2581  _runtime‚2581®M’JÇÛÁ & lr-AdamW/pg1‚0.0008077271925100221 & lr-AdamW/pg2‚0.0008077271925100221 trainer/global_step‚2199 % _timestamp‚1.7884665286307547e+09 _runtime‚2581.815771549 _step‚175Ë%„ÁÌÜÆ trainer/global_step‚2199 % _timestamp‚1.7884665286307547e+09 _runtime‚2581.815771549 _step‚175 & lr-AdamW/pg1‚0.0008077271925100221 & lr-AdamW/pg2‚0.0008077271925100221¯ùW32Ý- _wandbruntime‚2581  _runtime‚2581“#¾ânÞÅO N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _step‚176 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4350288900608 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40166624 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚93634624043520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚88341170668544 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40167306 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24509084 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚84190650240816 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚92691441566208 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚88341170668544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788466528631921e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59420852 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15658220 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316657664 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚2199 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41186555 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚98061025045504 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4426377061888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚88528602597241 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 100607407 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41186555 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4350522568192 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4338198376861 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4350288900608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41186557 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚92691693236736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24509086 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚88341152665600 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚84190633201456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚88341152665600 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚98060708387840 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚88528848617677 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4337969395785 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4426084344320 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59420852 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚88341170668544 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40167306 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24509084 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚84190650240816 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚92691441566208 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚88341170668544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788466528631921e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59420852 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15658220 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316657664 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚2199 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41186555 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚98061025045504 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4426377061888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚88528602597241 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 100607407 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41186555 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4350522568192 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4338198376861 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4350288900608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41186557 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚92691693236736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24509086 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚88341152665600 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚84190633201456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚88341152665600 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚98060708387840 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚88528848617677 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4337969395785°ÍÝI 2à- _wandbruntime‚2582  _runtime‚2582¦Añ"Ö&áÐM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352896 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚665 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _step‚177 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 100678744 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59473912 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100652374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59447544 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚88380438702592 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15664807 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚92732691629056 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚88380456705536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚93676331953664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100678744 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100652374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚667 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24519888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40184695 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40184028 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚92732943312896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15664142 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚98104389996544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚93676308013568 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41204830 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4428368395776 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚84228163558880 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41204832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4352252926464 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41204832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4340156866055 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 % _timestamp‚1.7884665296040556e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4352486607360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚88380438702592 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4339927884771 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚92732943312896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36~è‘üÛ&âÕM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41204832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4340156866055 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 % _timestamp‚1.7884665296040556e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4352486607360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚88380438702592 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4339927884771 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚92732943312896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352896 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚665 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _step‚177 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 100678744 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59473912 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100652374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59447544 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚88380438702592 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15664807 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚92732691629056 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚88380456705536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚93676331953664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100678744 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100652374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚667 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24519888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40184695 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40184028 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚92732943312896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15664142 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚98104389996544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚93676308013568 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41204830 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4428368395776 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚84228163558880 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41204832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4352252926464 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚94754210792960 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚90503058508949 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚94753959122432 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4435907436293 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _step‚180 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42100305 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41054968 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4448738274816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚94753959122432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60785146 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4436136417369 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4448504607232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4526158908416 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 # _timestamp‚1.78846658083849e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102859109 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4448504607232 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚90503304529385 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42100307 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60785146 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4525878773760 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚86067168112016 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚86067151072656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚100244701319680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25049186 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚95718822545920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42100305 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚95718846486016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚2634.023544503 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16005784 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280134656 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136“ gpÑ'ñËO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚100245005394432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 trainer/global_step‚2249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚94754210792960 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚90503058508949 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚94753959122432 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4435907436293 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _step‚180 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42100305 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41054968 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4448738274816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚94753959122432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60785146 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4436136417369 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4448504607232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4526158908416 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 # _timestamp‚1.78846658083849e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102859109 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4448504607232 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚90503304529385 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42100307 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60785146 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4525878773760 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚86067168112016 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚86067151072656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚100244701319680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25049186 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚95718822545920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42100305 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚95718846486016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚2634.023544503 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16005784 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280134656 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚95760530456064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4527876742144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚95760506515968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4528163154944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚90344758555136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 $ _timestamp‚1.788466581810667e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚90344740552192 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚94795209185280 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42118582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25059988 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚86104681430080 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚86104698469440 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41072460 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60785496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41073144 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102930446 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 trainer/global_step‚2249 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _step‚181 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102904076 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚94795460869120 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4450468633088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚90542793376003 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚90542547355359 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚94795460869120 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚100288693611008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4438094906563 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚90344740552192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4450468633088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25059986 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16012474 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582f¯x˜Ü&ôÖM ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚90344758555136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 $ _timestamp‚1.788466581810667e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚90344740552192 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚94795209185280 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42118582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25059988 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚86104681430080 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚86104698469440 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41072460 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60785496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41073144 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102930446 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 trainer/global_step‚2249 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _step‚181 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102904076 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚94795460869120 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4450468633088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚90542793376003 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚90542547355359 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚94795460869120 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚100288693611008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4438094906563 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚90344740552192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4450468633088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25059986 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16012474 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60811864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚684 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚100288383258112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4450702313984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚95760530456064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4527876742144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚95760506515968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4528163154944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582µ‹á;¥2õ- _wandbruntime‚2634  _runtime‚2634q¡›£ZöÔ .loss/FRt2/detection_loss‚2.068664983177937 /loss/FBt3/detection_loss‚0.5668358909244287 /loss/FBt4/detection_loss‚1.7938143478499518 0loss/FRt2/assignment_loss‚3.5473535502398454 0loss/FBt1/assignment_loss‚0.5509097333738787 1loss/SRqqt1/detection_loss‚1.8274760208231338 1loss/SRqqt2/detection_loss‚2.3871374068589044 /loss/FRt1/assignment_loss‚3.280350898251389 /loss/FRt4/assignment_loss‚4.092455953359604 2loss/SRqqt2/assignment_loss‚2.0980896980598054 1loss/SRqqt4/assignment_loss‚2.241477258503437 .loss/FRt1/detection_loss‚2.867432662934968 1loss/SRqqt3/detection_loss‚0.9366895466395047 2loss/SRqqt3/assignment_loss‚1.1505094375299372 1loss/FBt3/assignment_loss‚0.14837097177797964 -loss/FRt4/detection_loss‚3.60375091764662 % _timestamp‚1.7884665818133152e+09 0loss/FRt3/assignment_loss‚1.7123762127719349 2loss/SRqqt1/assignment_loss‚1.9146929510096287 1loss/FBt4/assignment_loss‚0.40572917483053317 .loss/FBt1/detection_loss‚2.144885500272115 /loss/FBt2/detection_loss‚2.5596937642377964 epoch‚1 _runtime‚ 2634.99803837 _step‚182 /loss/FRt3/detection_loss‚1.5571007128757766 &loss/total_loss‚46.293025554840746 1loss/FBt2/assignment_loss‚0.19594919411867842 1loss/SRqqt4/detection_loss‚2.6412787667729636 trainer/global_step‚2249/>€Í_÷Ù 0loss/FRt2/assignment_loss‚3.5473535502398454 0loss/FBt1/assignment_loss‚0.5509097333738787 1loss/SRqqt1/detection_loss‚1.8274760208231338 1loss/SRqqt2/detection_loss‚2.3871374068589044 /loss/FRt1/assignment_loss‚3.280350898251389 /loss/FRt4/assignment_loss‚4.092455953359604 2loss/SRqqt2/assignment_loss‚2.0980896980598054 1loss/SRqqt4/assignment_loss‚2.241477258503437 .loss/FRt1/detection_loss‚2.867432662934968 1loss/SRqqt3/detection_loss‚0.9366895466395047 2loss/SRqqt3/assignment_loss‚1.1505094375299372 1loss/FBt3/assignment_loss‚0.14837097177797964 -loss/FRt4/detection_loss‚3.60375091764662 % _timestamp‚1.7884665818133152e+09 0loss/FRt3/assignment_loss‚1.7123762127719349 2loss/SRqqt1/assignment_loss‚1.9146929510096287 1loss/FBt4/assignment_loss‚0.40572917483053317 .loss/FBt1/detection_loss‚2.144885500272115 /loss/FBt2/detection_loss‚2.5596937642377964 epoch‚1 _runtime‚ 2634.99803837 _step‚182 /loss/FRt3/detection_loss‚1.5571007128757766 &loss/total_loss‚46.293025554840746 1loss/FBt2/assignment_loss‚0.19594919411867842 1loss/SRqqt4/detection_loss‚2.6412787667729636 trainer/global_step‚2249 .loss/FRt2/detection_loss‚2.068664983177937 /loss/FBt3/detection_loss‚0.5668358909244287 /loss/FBt4/detection_loss‚1.7938143478499518¶ôH‘ø‚(:† š«çÔÞž‡ gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚102.658' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.6645 gpu.0.smClock‚1170 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-JØðoù‚(:ä š«çÔÛɈ disk.sda2.in‚3.625! disk.sdb2.out‚286.00732421875 proc.cpu.threads‚17 disk.dm-0.in‚528040.76171875 disk.sda1.in‚0 disk.sdb2.in‚ 3.49609375 network.sent‚ 4.396763e+06 disk.rbd2.out‚ 0.55859375 disk.nvme1n1.out‚ 21043.78125" disk.nvme0n1.out‚ 12951.7109375 disk.dm-0.out‚ 33995.4921875 disk.sda.in‚3.625 disk.md0p1.out‚ 276.1640625 disk.rbd0.out‚ 0.55859375% disk./.usageGB‚2973.4631271362305 disk.rbd2.in‚0! disk.sda2.out‚286.00732421875 disk.md0p1.in‚ 7.12109375 disk.md0.in‚ 7.12109375 disk.sdb.out‚286.00732421875 disk.sdb1.in‚0# disk.nvme0n1.in‚328074.76953125 disk.sda1.out‚0 cpu‚25.034570265648 network.recv‚885255" disk.nvme1n1.in‚199965.9921875 disk.sdb.in‚ 3.49609375 disk.sdb1.out‚0 disk.md0.out‚ 276.1640625 disk.rbd0.in‚0.015625$ memory_percent‚30.86252212524414( proc.memory.availableMB‚ 11327.484375) disk./.usagePercent‚83.13927352572959 disk.sda.out‚286.00732421875# proc.memory.rssMB‚ 3044.60546875* proc.memory.percent‚18.582797050476074=6Á ’ú‚(:‡ ©«çÔû鹇 gpu.0.gpu‚28.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚141.087' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.27175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û0ñtqû‚(:æ ©«çԇ싈 disk.sdb2.in‚ 3.49609375 disk.md0.out‚ 277.48828125 disk.rbd0.out‚ 0.55859375" disk.nvme1n1.in‚200167.3515625* proc.memory.percent‚18.582797050476074 disk.md0p1.in‚ 7.12109375 disk.rbd2.out‚ 0.55859375 disk.sda.out‚287.39208984375# proc.memory.rssMB‚ 3044.60546875 disk.md0p1.out‚ 277.48828125 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.availableMB‚11327.18359375 proc.cpu.threads‚17! disk.sdb2.out‚287.39208984375 disk.md0.in‚ 7.12109375 disk.sdb1.in‚0 disk.nvme0n1.in‚328227.5! disk.nvme0n1.out‚ 12952.359375 network.recv‚890528 disk.sda.in‚3.625 disk.sda2.in‚3.625 network.sent‚ 4.492842e+06# disk.nvme1n1.out‚21129.87890625 disk.sdb.out‚287.39208984375 disk.dm-0.in‚528394.8515625) disk./.usagePercent‚83.13801919787184% disk./.usageGB‚2973.4182662963867 disk.rbd0.in‚0.015625 disk.sdb.in‚ 3.49609375 disk.sdb1.out‚0! disk.sda2.out‚287.39208984375% memory_percent‚30.864357948303223 cpu‚25.03438674447934 disk.dm-0.out‚34082.23828125 disk.rbd2.in‚0ùµŠIœü‚(:‘ ¸«çÔׯ† gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦áè[ý‚(:Ð ¸«çÔŒˆ* proc.memory.percent‚18.582797050476074 proc.cpu.threads‚17 disk.sdb.in‚ 3.49609375 disk.sdb.out‚ 288.73046875 network.sent‚ 4.49668e+06) disk./.usagePercent‚83.14019293951648 disk.dm-0.in‚528394.8515625 disk.sdb2.out‚ 288.73046875# disk./.usageGB‚2973.49600982666 disk.sdb2.in‚ 3.49609375" disk.nvme1n1.in‚200167.3515625 disk.sda.in‚3.625 cpu‚25.034105383154966 disk.nvme0n1.out‚ 12952.59375 disk.sda2.in‚3.625 disk.rbd0.in‚0.015625 disk.sdb1.out‚0$ memory_percent‚30.86726665496826 disk.dm-0.out‚34169.02734375 disk.rbd2.in‚0# proc.memory.rssMB‚ 3044.60546875# disk.nvme1n1.out‚21216.43359375 disk.sda2.out‚ 288.73046875 disk.md0p1.out‚ 278.78125 network.recv‚894589 disk.nvme0n1.in‚328227.5 disk.sda1.out‚0 disk.md0p1.in‚ 7.12109375 disk.md0.out‚ 278.78125 disk.rbd0.out‚ 0.55859375 disk.rbd2.out‚ 0.55859375 disk.sda1.in‚0 disk.md0.in‚ 7.12109375 disk.sdb1.in‚0 disk.sda.out‚ 288.73046875* proc.memory.availableMB‚11326.70703125ÖÑD·’þ‚(:‡ Ç«çÔ²ÓÁ† gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚136.743' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.18575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Rê¬oÿ‚(:ä Ç«çÔÕ‹Õ‡$ disk./.usageGB‚2973.779769897461 disk.sda.out‚290.65185546875 disk.dm-0.in‚528394.8515625 disk.md0p1.in‚ 7.12109375 disk.md0.in‚ 7.12109375 disk.rbd2.out‚ 0.55859375* proc.memory.availableMB‚11327.06640625 disk.dm-0.out‚34267.42578125 disk.sda1.in‚0 disk.sda2.in‚3.625 disk.nvme1n1.out‚ 21314.78125* proc.memory.percent‚18.582797050476074 proc.cpu.threads‚17 disk.sdb1.in‚0# proc.memory.rssMB‚ 3044.60546875 network.sent‚ 4.501981e+06 network.recv‚898402 disk.nvme0n1.in‚328227.5 disk.sda1.out‚0! disk.sda2.out‚290.65185546875 disk.sdb.in‚ 3.49609375 disk.md0p1.out‚ 280.65234375 disk.md0.out‚ 280.65234375 disk.rbd0.out‚ 0.55859375 disk.sdb.out‚290.65185546875 disk.sda.in‚3.625# disk.nvme0n1.out‚12952.64453125% memory_percent‚30.865073204040527 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.sdb2.in‚ 3.49609375! disk.sdb2.out‚290.65185546875" disk.nvme1n1.in‚200167.3515625 cpu‚25.03401571417384) disk./.usagePercent‚83.14812698985888ÃD±2€- _wandbruntime‚2687  _runtime‚2687Oõ]ÇÁ _step‚183 & lr-AdamW/pg1‚0.0008042069694796327 & lr-AdamW/pg2‚0.0008042069694796327 trainer/global_step‚2299 % _timestamp‚1.7884666342234614e+09 _runtime‚2687.408537872]±Á-Ì‚Æ _runtime‚2687.408537872 _step‚183 & lr-AdamW/pg1‚0.0008042069694796327 & lr-AdamW/pg2‚0.0008042069694796327 trainer/global_step‚2299 % _timestamp‚1.7884666342234614e+09·`´Ýû2ƒ- _wandbruntime‚2687  _runtime‚2687ÒÑþ]’„ÈO GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 105137155 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43014057 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62123098 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚710 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 % _timestamp‚1.7884666342246995e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4546720313856 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚96816476678656 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4546720313856 _step‚184 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4546953981440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚97803044988416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚92477760441093 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41945069 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚102428710752768 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62096756 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚96816728349184 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16355783 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4533845476801 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4625689704448 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚96816728349184 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25589286 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 322949120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚92269756364800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43014057 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4534074457877 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚92269774367744 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚102429033701888 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _LDeviceStìvO<atsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚92269774367744 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚87943685983216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 trainer/global_step‚2299 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43014055 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62096756 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25589284 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚96816476678656 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62096756 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚96816728349184 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16355783 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4533845476801 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4625689704448 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚96816728349184 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25589286 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 322949120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚92269756364800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43014057 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4534074457877 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚92269774367744 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚102429033701888 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚92269774367744 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚87943685983216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 trainer/global_step‚2299 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43014055 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62096756 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25589284 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚96816476678656 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚97803044988416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚92477760441093 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41945069 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚102428710752768 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 trainer/global_step‚2299 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312450048 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚92309042401792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _runtime‚2688.386203912 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41962831 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4627974446080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4536032947071 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288509952 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4548918020608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4535803965787 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚96857726741504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4627685936128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚674 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚97844728958464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62123448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41962157 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43032332 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚102472390954496 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 _step‚185 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚87981199301280 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43032330 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62149816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105182148 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 105155778 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚87981216340640 % _timestamp‚1.7884666352004201e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43032330 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25600086 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚672 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 105155778 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚92517249287711 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚96857978425344 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4548918020608 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚96857978425344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43032332 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚92309060404736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚92309060404736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4548684339712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚102472703404544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62149816 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚97844705018368 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0Áò›òÝ&ˆ×M ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105182148 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 105155778 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚87981216340640 % _timestamp‚1.7884666352004201e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43032330 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25600086 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚672 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 105155778 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚92517249287711 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚96857978425344 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4548918020608 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚96857978425344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43032332 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚92309060404736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚92309060404736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4548684339712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚102472703404544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62149816 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚97844705018368 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62123448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25600088 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚92309042401792 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 trainer/global_step‚2299 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312450048 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚92309042401792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _runtime‚2688.386203912 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41962831 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4627974446080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4536032947071 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288509952 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4548918020608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4535803965787 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚96857726741504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4627685936128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚674 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚97844728958464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62123448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41962157 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43032332 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚102472390954496 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 _step‚185 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚87981199301280 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43032330 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62149816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0¹©ìÛs2‰- _wandbruntime‚2688  _runtime‚2688g&æ%§ŠÑ 0loss/FRt3/assignment_loss‚1.3946284929911295 .loss/FRt2/detection_loss‚2.213136149154223 /loss/FBt4/detection_loss‚1.7343687065269635 % _timestamp‚1.7884666352035482e+09 0loss/FBt2/assignment_loss‚0.8110382672979246 .loss/FBt1/detection_loss‚2.108735316991806 /loss/FBt3/detection_loss‚0.6071878859316403 /loss/FRt2/assignment_loss‚4.155413934525023 2loss/SRqqt3/assignment_loss‚1˜G–D°.0029011129373961 0loss/FBt3/assignment_loss‚0.2497625778630127 0loss/SRqqt1/detection_loss‚1.996678571117685 1loss/SRqqt4/detection_loss‚2.7032720781862736 .loss/FRt1/detection_loss‚3.076687303456393 /loss/FRt4/detection_loss‚2.8570104082425436 1loss/SRqqt3/detection_loss‚1.1433263344339806 epoch‚1 trainer/global_step‚2299 _runtime‚2688.388317842 /loss/FRt4/assignment_loss‚4.832365489006042 /loss/FRt1/assignment_loss‚3.106575472788377 2loss/SRqqt2/assignment_loss‚2.3548711207177906 0loss/FBt1/assignment_loss‚0.5481720147627612 1loss/FBt4/assignment_loss‚0.32423180382211014 /loss/FRt3/detection_loss‚1.5325783479213715 _step‚186 2loss/SRqqt4/assignment_loss‚2.4018938317894936 .loss/FBt2/detection_loss‚2.098244200734531 1loss/SRqqt1/assignment_loss‚1.359335357006243 1loss/SRqqt2/detection_loss‚2.4455861465798483 %loss/total_loss‚47.05800092478456äõwÂ\‹Ö trainer/global_step‚2299 _runtime‚2688.388317842 /loss/FRt4/assignment_loss‚4.832365489006042 /loss/FRt1/assignment_loss‚3.106575472788377 2loss/SRqqt2/assignment_loss‚2.3548711207177906 0loss/FBt1/assignment_loss‚0.5481720147627612 1loss/FBt4/assignment_loss‚0.32423180382211014 /loss/FRt3/detection_loss‚1.5325783479213715 _step‚186 2loss/SRqqt4/assignment_loss‚2.4018938317894936 .loss/FBt2/detection_loss‚2.098244200734531 1loss/SRqqt1/assignment_loss‚1.359335357006243 1loss/SRqqt2/detection_loss‚2.4455861465798483 %loss/total_loss‚47.05800092478456 0loss/FRt3/assignment_loss‚1.3946284929911295 .loss/FRt2/detection_loss‚2.213136149154223 /loss/FBt4/detection_loss‚1.7343687065269635 % _timestamp‚1.7884666352035482e+09 0loss/FBt2/assignment_loss‚0.8110382672979246 .loss/FBt1/detection_loss‚2.108735316991806 /loss/FBt3/detection_loss‚0.6071878859316403 /loss/FRt2/assignment_loss‚4.155413934525023 2loss/SRqqt3/assignment_loss‚1.0029011129373961 0loss/FBt3/assignment_loss‚0.2497625778630127 0loss/SRqqt1/detection_loss‚1.996678571117685 1loss/SRqqt4/detection_loss‚2.7032720781862736 .loss/FRt1/detection_loss‚3.076687303456393 /loss/FRt4/detection_loss‚2.8570104082425436 1loss/SRqqt3/detection_loss‚1.1433263344339806 epoch‚1ºHÇÍ’Œ‚(:‡ Ö«çÔ¿®é† gpu.0.gpu‚27.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚120.927' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.23175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00ªGm‚(:â Ö«çÔÕͳ‡) proc.memory.availableMB‚ 11327.0234375 disk.sda1.in‚0 disk.sdb2.in‚ 3.49609375 disk.sdb.out‚292.33740234375( disk./.usagePercent‚83.1612599945185 disk.dm-0.in‚528394.8515625 disk.sda1.out‚0 disk.sda2.in‚3.625! disk.sda2.out‚292.33740234375 disk.rbd0.out‚ 0.55859375! disk.sdb2.out‚292.33740234375 disk.rbd0.in‚0.015625 cpu‚25.03383403640558 disk.sdb1.out‚0 disk.md0p1.in‚ 7.12109375 disk.md0p1.out‚ 282.26953125 disk.md0.out‚ 282.26953125 disk.rbd2.out‚ 0.55859375" disk.nvme1n1.in‚200167.3515625 proc.cpu.threads‚17 network.sent‚ 4.508314e+06 disk.sda.in‚3.625# disk.nvme0n1.out‚12953.04296875 disk.md0.in‚ 7.12109375 disk.rbd2.in‚0" disk.nvme1n1.out‚ 21573.2109375 disk.nvme0n1.in‚328227.5# proc.memory.rssMB‚ 3044.60546875* proc.memory.percent‚18.582797050476074 network.recv‚903129 disk.sda.out‚292.33740234375 disk.sdb.in‚ 3.49609375# disk./.usageGB‚2974.24946975708 disk.sdb1.in‚0 disk.dm-0.out‚34526.25390625$ memory_percent‚30.86533546447754öiŽ‚(:… å«çÔ¶§ã‡ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚123.712' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.928 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~2]¹l‚(:á å«çÔª«Éˆ network.recv‚907548 disk.md0p1.in‚ 7.12109375# disk.nvme0n1.out‚12953.09765625 disk.dm-0.in‚528394.8515625 disk.dm-0.out‚ 34805.890625 disk.rbd2.in‚0 disk.sdb.out‚293.7451171875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚293.7451171875 disk.md0.in‚ 7.12109375 disk.md0.out‚ 283.6171875 disk.rbd0.out‚ 0.55859375 disk.nvme0n1.in‚328227.5 disk.sda.out‚293.7451171875 cpu‚25.033718080398586 network.sent‚ 4.614398e+06) disk./.usagePercent‚83.16826428500241% disk./.usageGB‚2974.4999771118164 disk.md0p1.out‚ 283.6171875 disk.sdb2.out‚293.7451171875 disk.rbd2.out‚ 0.55859375# disk.nvme1n1.out‚21852.79296875 disk.sda.in‚3.625 disk.sdb1.in‚0% memory_percent‚30.866265296936035* proc.memory.availableMB‚11326.87109375# proc.memory.rssMB‚ 3044.60546875 disk.rbd0.in‚0.015625" disk.nvme1n1.in‚200167.3515625 disk.sdb.in‚ 3.49609375* proc.memory.percent‚18.582797050476074 proc.cpu.threads‚17 disk.sda2.in‚3.625 disk.sdb2.in‚ 3.49609375 disk.sdb1.out‚0çŸ&µ‘‚(:† ô«çÔÔÇ܇ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.966' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.9915 gpu.0.smClock‚1245 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•¯\`h‘‚(:Ý ô«çÔ¶ÎÈ disk.rbd2.out‚ 0.55859375 disk.rbd2.in‚0" disk./.usageGB‚2975.0009765625 disk.sdb1.out‚0 disk.md0p1.out‚ 285.46875* proc.memory.availableMB‚11326.85546875 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda2.in‚3.625! disk.sda2.out‚295.65673828125" disk.nvme1n1.in‚200167.3515625% memory_percent‚30.866360664367676 disk.dm-0.out‚34864.96484375 disk.rbd0.in‚0.015625 cpu‚25.033716805279898 disk.md0.in‚ 7.12109375 network.recv‚911894 disk.sdb.in‚ 3.49609375 disk.sdb.out‚295.65673828125 disk.sda.in‚3.625 disk.md0p1.in‚ 7.12109375 disk.sdb2.in‚ 3.49609375! disk.sdb2.out‚295.65673828125 disk.sda.out‚295.65673828125 disk.nvme0n1.in‚328227.5 disk.sda1.in‚0 disk.rbd0.out‚ 0.55859375# proc.memory.rssMB‚ 3044.61328125* proc.memory.percent‚18.582844734191895 network.sent‚ 4.620432e+06# disk.nvme0n1.out‚12953.24609375 disk.md0.out‚ 285.46875 disk.nvme1n1.out‚ 21911.71875) disk./.usagePercent‚83.18227243932807 disk.sdb1.in‚0 disk.dm-0.in‚528394.85156258N2’- _wandbruntime‚2740  _runtime‚2740œ¥TÆ“À & lr-AdamW/pg1‚0.0008023241380266154 & lr-AdamW/pg2‚0.0008023241380266154 trainer/global_step‚2349 $ _timestamp‚1.788466687767333e+09 _runtime‚2740.952270128 _step‚187Ž/Þ+˔Š& lr-AdamW/pg1‚0.0008023241380266154 & lr-AdamW/pg2‚0.0008023241380266154 trainer/global_step‚2349 $ _timestamp‚1.788466687767333e+09 _runtime‚2740.952270128 _step‚187»óù$2•- _wandbruntime‚2740  _runtime‚27402 Í'–ÇO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4644936020480 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16704034 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚686 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4631783517309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚94234058214400 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚104613023255552 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107362513 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚94234076217344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43927805 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚98879245905408 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚94452216352801 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚98878994234880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107388857 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306171904 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4725779764736 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚104612717083648 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4725497532928 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4644936020480 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚94234058214400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4645169688064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚98879245905408 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚94451970332365 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚89820186815056 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚98878994234880 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26129386 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16703348 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42832732 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚94452216352801 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚98878994234880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107388857 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306171904 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4725779764736 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚104612717083648 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4725497532928 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4644936020480 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚94234058214400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4645169688064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚98879245905408 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚94451970332365 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚89820186815056 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚98878994234880 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26129386 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16703348 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42832732 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4631783517309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚94234058214400 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚104613023255552 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107362513 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚94234076217344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43927805 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496¼Ñ¨^2˜- _wandbruntime‚2742  _runtime‚2742—£süÊ™ÒM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43946082 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4647133727232 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚98920244297728 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4633742006295 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 107407480 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107407480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43946080 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294801408 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚94273362254336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42850945 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107433850 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63461400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4646900046336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚98920495981568 _runtime‚2742.137769042 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚89857734211840 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63461400 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 OGDeviceStatsMonitor.on_train_batch_end/inacti0ve_split.small_pool.current‚700 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16710057 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4646900046336 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚94273344251392 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4633970987579 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 % _timestamp‚1.7884666889522488e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4647133727232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚94491705199419 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42850243 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚99928903520768 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43946082 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4727486703104 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0›IŽÝ&š×M H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 107407480 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107407480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43946080 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294801408 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚94273362254336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42850945 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107433850 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63461400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4646900046336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚98920495981568 _runtime‚2742.137769042 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚89857734211840 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63461400 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚700 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16710057 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4646900046336 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚94273344251392 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4633970987579 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 % _timestamp‚1.7884666889522488e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4647133727232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚94491705199419 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42850243 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚99928903520768 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43946082 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4727486703104 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43946082 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4647133727232 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚98920244297728 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4633742006295 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0½¥m12›- _wandbruntime‚2742  _runtime‚2742‚NøZœÔ 0loss/FRt3/assignment_loss‚1.6741750422451231 0loss/FBt1/assignment_loss‚0.2860247981515272 1loss/FBt2/assignment_loss‚0.30552892149191596 0loss/SRqqt2/detection_loss‚2.535106151192277 /loss/FBt3/detection_loss‚0.4927003019422825 _step‚190 /loss/FRt1/assignment_loss‚3.336323943208246 2loss/SRqqt4/assignment_loss‚2.0351282358169556 0loss/FBt4/assignment_loss‚0.3332775784656285 1loss/SRqqt4/detection_loss‚3.2535977626548096 .loss/FBt4/detection_loss‚2.207850795525771 %loss/total_loss‚46.65929873776392 2loss/SRqqt3/assignment_loss‚1.2463084859145885 1loss/FBt3/assignment_loss‚0.08254666035642343 0loss/SRqqt1/detection_loss‚2.403055436683424 /loss/FBt2/detection_loss‚2.6381345556841955 % _timestamp‚1.7884666889540238e+09 /loss/FRt2/assignment_loss‚4.188633697188419 2loss/SRqqt1/assignment_loss‚1.3598023743340464 2loss/SRqqt2/assignment_loss‚1.8868133182878848 _runtime‚2742.138749616 trainer/global_step‚2349 /loss/FRt2/detection_loss‚2.1728909241804932 epoch‚1 0loss/FRt4/assignment_loss‚3.9140724905075563 .loss/FRt3/detection_loss‚1.456778972254445 /loss/FRt4/detection_loss‚2.9828055875916637 /loss/FRt1/detection_loss‚2.7382558952359592 1loss/SRqqt3/detection_loss‚0.9472870763877164 .loss/FBt1/detection_loss‚2.182199732462565ëâ¢_Ù /loss/FRt1/detection_loss‚2.7382558952359592 1loss/SRqqt3/detection_loss‚0.9472870763877164 .loss/FBt1/detection_loss‚2.182199732462565 0loss/FRt3/assignment_loss‚1.6741750422451231 0loss/FBt1/assignment_loss‚0.2860247981515272 1loss/FBt2/assignment_loss‚0.30552892149191596 0loss/SRqqt2/detection_loss‚2.535106151192277 /loss/FBt3/detection_loss‚0.4927003019422825 _step‚190 /loss/FRt1/assignment_loss‚3.336323943208246 2loss/SRqqt4/assignment_loss‚2.0351282358169556 0loss/FBt4/assignment_loss‚0.3332775784656285 1loss/SRqqt4/detection_loss‚3.2535977626548096 .loss/FBt4/detection_loss‚2.207850795525771 %loss/total_loss‚46.65929873776392 2loss/SRqqt3/assignment_loss‚1.2463084859145885 1loss/FBt3/assignment_loss‚0.08254666035642343 0loss/SRqqt1/detection_loss‚2.403055436683424 /loss/FBt2/detection_loss‚2.6381345556841955 % _timestamp‚1.7884666889540238e+09 /loss/FRt2/assignment_loss‚4.188633697188419 2loss/SRqqt1/assignment_loss‚1.3598023743340464 2loss/SRqqt2/assignment_loss‚1.8868133182878848 _runtime‚2742.138749616 trainer/global_step‚2349 /loss/FRt2/detection_loss‚2.1728909241804932 epoch‚1 0loss/FRt4/assignment_loss‚3.9140724905075563 .loss/FRt3/detection_loss‚1.456778972254445 /loss/FRt4/detection_loss‚2.9828055875916637¾3uµDž‚(:… ƒ¬çÔö›³‡ gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚139.868' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.967 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤€HkhŸ‚(:Ý ƒ¬çÔ£üü‡ disk.sda.in‚3.625# disk.nvme0n1.in‚328227.71484375 disk.dm-0.out‚ 35211.015625 disk.md0p1.out‚ 287.140625 disk.sdb2.in‚ 3.49609375% memory_percent‚30.864787101745605# proc.memory.rssMB‚ 3044.61328125) disk./.usagePercent‚83.18221601590659 disk.sda2.in‚3.625 disk.sdb2.out‚ 297.392578125 disk.nvme1n1.out‚ 22257.71875 network.recv‚915407 disk.sdb1.in‚0 disk.dm-0.in‚528395.06640625 disk.md0.in‚ 7.12109375 disk.rbd0.out‚ 0.55859375" disk.nvme1n1.in‚200167.3515625% disk./.usageGB‚2974.9989585876465! disk.nvme0n1.out‚ 12953.296875 disk.md0p1.in‚ 7.12109375 disk.rbd0.in‚0.015625* proc.memory.availableMB‚11327.11328125 network.sent‚ 4.625753e+06 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda2.out‚ 297.392578125 disk.md0.out‚ 287.140625 disk.rbd2.in‚0 disk.sdb.in‚ 3.49609375 disk.sdb1.out‚0 disk.sda.out‚ 297.392578125 disk.rbd2.out‚ 0.55859375* proc.memory.percent‚18.582844734191895 disk.sdb.out‚ 297.392578125 cpu‚25.03354519319198"óŽY‘ ‚(:† ’¬çÔЭ߇ gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚124.994' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.2485 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>¦CZs¡‚(:è ’¬çÔϢLj* proc.memory.availableMB‚11326.83203125 cpu‚25.033533251558406 network.recv‚921045 disk.sdb2.out‚ 299.103515625 disk.rbd0.out‚ 0.55859375 disk.sda1.out‚0 disk.sdb2.in‚ 3.49609375 disk.md0.out‚ 288.78515625 proc.cpu.threads‚17 disk.sdb.out‚ 299.103515625 disk.sdb1.out‚0 disk.dm-0.out‚35432.51953125 disk.sda1.in‚0* proc.memory.percent‚18.582844734191895 network.sent‚ 4.733122e+06% disk./.usageGB‚2974.9989585876465 disk.sda.out‚ 299.103515625 disk.md0.in‚ 7.12109375 disk.rbd2.in‚0% memory_percent‚30.866503715515137# proc.memory.rssMB‚ 3044.61328125 disk.sdb1.in‚0" disk.nvme1n1.in‚200167.3515625" disk.nvme1n1.out‚ 22478.9453125# disk.nvme0n1.in‚328227.71484375 disk.sda2.out‚ 299.103515625) disk./.usagePercent‚83.18221601590659 disk.sdb.in‚ 3.49609375 disk.sda.in‚3.625# disk.nvme0n1.out‚12953.57421875 disk.sda2.in‚3.625 disk.md0p1.in‚ 7.12109375 disk.md0p1.out‚ 288.78515625 disk.rbd2.out‚ 0.55859375 disk.dm-0.in‚528395.06640625 disk.rbd0.in‚0.015625ñßB¾d¢‚(:Ù ¡¬çÔƶ´‰% disk./.usageGB‚2974.9989585876465 disk.sdb2.in‚ 3.49609375# disk.nvme0n1.in‚328227.71484375 disk.dm-0.out‚ 35807.90625% memory_percent‚30.867958068847656) disk./.usagePercent‚83.18221601590659 disk.rbd2.in‚0 disk.sda1.in‚0 disk.md0p1.out‚ 290.21875 proc.cpu.threads‚17 network.sent‚ 4.738915e+06 disk.md0.out‚ 290.21875 disk.sdb1.out‚0 cpu‚25.033516199952487 network.recv‚925342" disk.nvme1n1.in‚200167.3515625 disk.sda.in‚3.625 disk.sda.out‚300.5966796875* proc.memory.percent‚18.582844734191895 disk.sdb2.out‚300.5966796875 disk.dm-0.in‚528395.06640625 disk.sda2.out‚300.5966796875' proc.memory.availableMB‚ 11326.59375 disk.rbd0.out‚ 0.55859375 disk.sdb1.in‚0 disk.nvme0n1.out‚ 12953.625 disk.sda1.out‚0 disk.sda2.in‚3.625 disk.nvme1n1.out‚ 22854.28125 disk.sdb.in‚ 3.49609375 disk.md0p1.in‚ 7.12109375# proc.memory.rssMB‚ 3044.61328125 disk.md0.in‚ 7.12109375 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 0.55859375 disk.sdb.out‚300.5966796875hT#Ÿ’£‚(:‡ ¡¬çÔâáЈ gpu.0.gpu‚10.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚137.693' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.42325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðÝèÅ’¤‚(:‡ °¬çÔÂ乇 gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚137.693' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.42325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0WŒæÜs¥‚(:è °¬çÔû‚ᇠnetwork.sent‚ 4.747111e+06% disk./.usageGB‚2974.9989585876465 disk.sda1.in‚0* proc.memory.percent‚18.582844734191895 disk.sdb2.in‚ 3.49609375 disk.rbd0.out‚ 0.55859375 disk.md0p1.in‚ 7.12109375 disk.md0.out‚ 292.12109375 disk.sdb.out‚302.5673828125 disk.sdb1.in‚0 disk.sdb1.out‚0# disk.nvme0n1.in‚328227.71484375 disk.dm-0.in‚528395.06640625 disk.rbd0.in‚0.015625* proc.memory.availableMB‚11326.57421875 cpu‚25.033545850559825 disk.md0.in‚ 7.12109375" disk.nvme1n1.in‚200167.3515625 disk.nvme1n1.out‚ 23063.40625 disk.sda.in‚3.625# proc.memory.rssMB‚ 3044.61328125 disk.sdb2.out‚302.5673828125 disk.sda.out‚302.5673828125% memory_percent‚30.868077278137207 network.recv‚929468) disk./.usagePercent‚83.18221601590659 disk.dm-0.out‚ 36017.1328125 disk.sda1.out‚0 disk.sda2.in‚3.625 disk.md0p1.out‚ 292.12109375 disk.rbd2.out‚ 0.55859375 disk.sdb.in‚ 3.49609375 disk.sda2.out‚302.5673828125 disk.rbd2.in‚0" disk.nvme0n1.out‚ 12953.7265625 proc.cpu.threads‚17o/S^2¦- _wandbruntime‚2797  _runtime‚2797'<ò<ǧÁ % _timestamp‚1.7884667439455817e+09 _runtime‚2797.130520216 _step‚191 & lr-AdamW/pg1‚0.0008003600107794893 & lr-AdamW/pg2‚0.0008003600107794893 trainer/global_step‚2399ÂEþÂÌ¨Æ & lr-AdamW/pg2‚0.0008003600107794893 trainer/global_step‚2399 % _timestamp‚1.7884667439455817e+09 _runtime‚2797.130520216 _step‚191 & lr-AdamW/pg1‚0.0008003600107794893¿ŠƒÞå2©- _wandbruntime‚2797  _runtime‚27970*%¸¾ªÎO XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚96198360064000 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚96426672264509 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26669484 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4743385394688 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109640559 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚100941763461632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7884667439466288e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚101971441993216 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚91696704686256 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚96198360064000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43722267 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4825654868480 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚662 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316657664 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚100941511791104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109614215 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚96426426244073 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64799002 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚100941763461632 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚664 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4825362150912 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _runtime‚2797.131671371 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚100941511791104 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17052781 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚106797096861696 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26669486 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17052119 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44841557 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4743151727104 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚106796780204032 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582¥ŸÐÙ'«ÓO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4743385394688 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109640559 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚100941763461632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7884667439466288e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚101971441993216 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚91696704686256 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚96198360064000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43722267 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4825654868480 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚662 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316657664 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚100941511791104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109614215 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚96426426244073 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64799002 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚100941763461632 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚664 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4825362150912 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _runtime‚2797.131671371 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚100941511791104 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17052781 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚106797096861696 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26669486 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17052119 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44841557 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4743151727104 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚106796780204032 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚96198360064000 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚96426672264509 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26669484 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608À¦wUÖ2¬- _wandbruntime‚2798  _runtime‚2798NKo“Þ&­ØM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚685 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚106840769081856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4731909028087 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚91734235043680 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚96237664103936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4745115752960 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43740310 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4827643118592 _step‚193 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64799352 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚96237664103936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚102013125963264 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚2399 _runtime‚2798.295166685 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44859832 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884667451093202e+09 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚100983013537792 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64799352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚687 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412800 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109685552 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚96466161111127 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚91734252083040 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4745349433856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44859832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚96465915090483 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚100983013537792 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4745115752960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44859830 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17060022 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4827356705792 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚100982761853952 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64825720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26680288 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26680286 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17059337 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64825720 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4745349433856 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚102013102023168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109685552 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109659182 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚96237646100992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚106840458728960 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44859830 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4731680046803 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43739623 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496Њiã&®ÝM XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚106840458728960 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44859830 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4731680046803 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43739623 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚685 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚106840769081856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4731909028087 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚91734235043680 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚96237664103936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4745115752960 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43740310 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4827643118592 _step‚193 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64799352 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚96237664103936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚102013125963264 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚2399 _runtime‚2798.295166685 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44859832 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884667451093202e+09 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚100983013537792 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64799352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚687 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412800 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109685552 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚96466161111127 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚91734252083040 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4745349433856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44859832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚96465915090483 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚100983013537792 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4745115752960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44859830 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17060022 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4827356705792 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚100982761853952 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64825720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26680288 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26680286 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17059337 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64825720 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4745349433856 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚102013102023168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109685552 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109659182 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚96237646100992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2ÁÒÌœ2¯- _wandbruntime‚2798  _runtime‚2798îö(»°Ö /loss/FRt2/detection_loss‚2.2973956709130383 1loss/SRqqt3/detection_loss‚0.6987405272625518 %loss/total_loss‚46.31861448779672 0loss/FRt3/assignment_loss‚1.2034886500291657 0loss/FRt4/assignment_loss‚3.5112670759359994 2loss/SRqqt4/assignment_loss‚2.5155134855045214 _step‚194 0loss/FBt2/assignment_loss‚0.2977512340164847 .loss/FRt3/detection_loss‚1.583428702987077 0loss/FRt1/assignment_loss‚4Y£*„¡.1600214930681085 2loss/SRqqt2/assignment_loss‚1.9776417095871532 2loss/SRqqt3/assignment_loss‚0.9868885087127095 .loss/FBt2/detection_loss‚2.504911389615801 /loss/FBt3/detection_loss‚0.6579906608909368 % _timestamp‚1.7884667451113746e+09 2loss/SRqqt1/assignment_loss‚1.7239956806392172 .loss/FRt1/detection_loss‚3.012979120016098 trainer/global_step‚2399 1loss/FBt1/assignment_loss‚0.35602539211588036 1loss/FBt4/assignment_loss‚0.32735082280955147 /loss/FRt4/detection_loss‚3.0111707051595054 /loss/FBt1/detection_loss‚1.9158751289049785 /loss/FBt4/detection_loss‚2.7439060466630116 _runtime‚2798.296131209 1loss/SRqqt1/detection_loss‚2.1153240481088327 1loss/SRqqt4/detection_loss‚2.2542941777242556 -loss/FRt2/assignment_loss‚3.9463146912015 2loss/FBt3/assignment_loss‚0.049882977353525086 0loss/SRqqt2/detection_loss‚2.466456588576822 epoch‚1GvÇa±Û /loss/FRt2/detection_loss‚2.2973956709130383 1loss/SRqqt3/detection_loss‚0.6987405272625518 %loss/total_loss‚46.31861448779672 0loss/FRt3/assignment_loss‚1.2034886500291657 0loss/FRt4/assignment_loss‚3.5112670759359994 2loss/SRqqt4/assignment_loss‚2.5155134855045214 _step‚194 0loss/FBt2/assignment_loss‚0.2977512340164847 .loss/FRt3/detection_loss‚1.583428702987077 0loss/FRt1/assignment_loss‚4.1600214930681085 2loss/SRqqt2/assignment_loss‚1.9776417095871532 2loss/SRqqt3/assignment_loss‚0.9868885087127095 .loss/FBt2/detection_loss‚2.504911389615801 /loss/FBt3/detection_loss‚0.6579906608909368 % _timestamp‚1.7884667451113746e+09 2loss/SRqqt1/assignment_loss‚1.7239956806392172 .loss/FRt1/detection_loss‚3.012979120016098 trainer/global_step‚2399 1loss/FBt1/assignment_loss‚0.35602539211588036 1loss/FBt4/assignment_loss‚0.32735082280955147 /loss/FRt4/detection_loss‚3.0111707051595054 /loss/FBt1/detection_loss‚1.9158751289049785 /loss/FBt4/detection_loss‚2.7439060466630116 _runtime‚2798.296131209 1loss/SRqqt1/detection_loss‚2.1153240481088327 1loss/SRqqt4/detection_loss‚2.2542941777242556 -loss/FRt2/assignment_loss‚3.9463146912015 2loss/FBt3/assignment_loss‚0.049882977353525086 0loss/SRqqt2/detection_loss‚2.466456588576822 epoch‚1·G‡a‘²‚(:† ¿¬çÔ±ŽÌ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.966' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.9915 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w3—t³‚(:é ¿¬çÔ®ÿÆ’ disk.dm-0.out‚ 36211.890625 disk.sda1.in‚0 disk.sdb2.out‚304.0732421875# proc.memory.rssMB‚ 3044.61328125 cpu‚25.033610239071066 disk.sdb.out‚304.0732421875 disk.sda.out‚304.0732421875% disk./.usageGB‚2974.9990158081055 disk.sda2.in‚3.625 disk.sda2.out‚304.0732421875 disk.sdb2.in‚ 3.49609375 disk.md0.out‚ 293.56640625 disk.rbd2.in‚0) disk./.usagePercent‚83.18221761581457 disk.dm-0.in‚528395.06640625 disk.rbd0.out‚ 0.55859375 disk.rbd2.out‚ 0.55859375 disk.sdb1.in‚0 disk.sda.in‚3.625" disk.nvme1n1.in‚200167.3515625# disk.nvme1n1.out‚23258.07421875# disk.nvme0n1.in‚328227.71484375# disk.nvme0n1.out‚12953.81640625 proc.cpu.threads‚17 network.sent‚ 4.74929e+06 disk.sda1.out‚0 disk.md0p1.in‚ 7.12109375 disk.md0.in‚ 7.12109375 disk.rbd0.in‚0.015625 disk.sdb.in‚ 3.49609375 disk.sdb1.out‚0$ memory_percent‚30.86550235748291* proc.memory.availableMB‚11326.99609375* proc.memory.percent‚18.582844734191895 network.recv‚932318 disk.md0p1.out‚ 293.56640625N̵\´‚(:… άçÔ…첇 gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚149.782' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚37.4455 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0PzVÓrµ‚(:ç άçÔèõˆ! disk.sda2.out‚305.60400390625 network.recv‚938005 disk.rbd0.in‚0.015625 disk.sdb.out‚305.60400390625! disk.sdb2.out‚305.60400390625" disk.nvme1n1.out‚ 23509.4609375 disk.sda1.out‚0 disk.sdb2.in‚ 3.49609375 disk.dm-0.out‚36463.41796875 disk.sda1.in‚0# proc.memory.rssMB‚ 3044.61328125 cpu‚25.03493860626434 disk.md0.in‚ 7.12109375 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.sda.in‚3.625 disk.md0p1.in‚ 7.12109375) disk./.usagePercent‚83.18221601590659% disk./.usageGB‚2974.9989585876465" disk.nvme1n1.in‚200167.3515625 disk.sdb1.in‚0# disk.nvme0n1.out‚12953.95703125$ memory_percent‚30.86540699005127 network.sent‚ 4.85913e+06 disk.md0.out‚ 295.03125 disk.rbd0.out‚ 0.55859375 disk.sdb.in‚ 3.49609375# disk.nvme0n1.in‚328227.71484375 proc.cpu.threads‚17 disk.rbd2.out‚ 0.55859375 disk.sda.out‚305.60400390625 disk.dm-0.in‚528395.06640625* proc.memory.availableMB‚11327.01171875* proc.memory.percent‚18.582844734191895 disk.sda2.in‚3.625 disk.md0p1.out‚ 295.03125¢ð¶Ð’¶‚(:‡ ݬçÔ¶¸Œ‡ gpu.0.gpu‚36.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q Mpr·‚(:ç ݬçÔƒ·¢ˆ# disk.nvme0n1.in‚328227.71484375 disk.sdb.in‚ 3.49609375 disk.sdb1.out‚0 network.sent‚ 4.865036e+06 disk.md0.in‚ 7.12109375* proc.memory.availableMB‚11326.98828125 cpu‚25.05654317800038 disk.rbd0.in‚0.015625" disk.nvme1n1.in‚200167.3515625" disk.nvme0n1.out‚ 12954.0234375 disk.dm-0.out‚ 36713.78125 disk.sdb2.in‚ 3.49609375 disk.sda.out‚307.55419921875# proc.memory.rssMB‚ 3044.61328125 proc.cpu.threads‚17) disk./.usagePercent‚83.16739148186805 disk.md0p1.in‚ 7.12109375 disk.rbd0.out‚ 0.55859375 disk.rbd2.in‚0 disk.sdb.out‚307.55419921875 disk.sda.in‚3.625* proc.memory.percent‚18.582844734191895 disk.sda2.in‚3.625! disk.sda2.out‚307.55419921875$ disk./.usageGB‚2974.468761444092 disk.rbd2.out‚ 0.55859375" disk.nvme1n1.out‚ 23759.7578125 disk.sdb1.in‚0$ memory_percent‚30.86555004119873 network.recv‚942073 disk.dm-0.in‚528395.06640625 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0.out‚ 296.9140625 disk.md0p1.out‚ 296.9140625! disk.sdb2.out‚307.55419921875•í¥Ëœ¸‚(:‘ ì¬çÔŸõƒ† gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚107.611' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚26.902750000000005 gpu.0.smClock‚1245 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÖDÁ˜o¹‚(:ä ì¬çÔ’¿ˆ$ disk./.usageGB‚2974.467674255371# disk.nvme1n1.out‚24008.68359375 disk.sda.in‚3.625 proc.cpu.threads‚17! disk.sdb2.out‚308.88330078125 disk.md0.out‚298.1875 disk.rbd2.in‚0$ memory_percent‚30.86552619934082 network.recv‚946207 disk.sdb1.out‚0 disk.sda2.in‚3.625# proc.memory.rssMB‚ 3044.61328125 cpu‚25.077227361527722 disk.sdb2.in‚ 3.49609375 disk.rbd0.in‚0.015625 disk.sdb.in‚ 3.49609375 disk.sdb.out‚308.88330078125# disk.nvme0n1.in‚328227.71484375( disk./.usagePercent‚83.1673610836164" disk.nvme1n1.in‚200167.3515625 disk.dm-0.in‚528395.06640625 network.sent‚ 4.871083e+06 disk.md0p1.out‚298.1875 disk.rbd0.out‚ 0.55859375 disk.sda.out‚308.88330078125 disk.md0p1.in‚ 7.12109375 disk.md0.in‚ 7.12109375 disk.rbd2.out‚ 0.55859375# disk.nvme0n1.out‚12954.19921875 disk.sda1.in‚0! disk.sda2.out‚308.88330078125) proc.memory.availableMB‚ 11326.9921875 disk.sdb1.in‚0 disk.dm-0.out‚ 36962.8828125 disk.sda1.out‚0* proc.memory.percent‚18.582844734191895Ò¹Ú‘º‚(:† û¬çÔ’À‰ˆ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó½‡t»‚(:é û¬çÔñûŽ‰ disk.sdb2.out‚310.3955078125 cpu‚25.09812092171035 network.recv‚949556 disk.sdb1.out‚0 disk.sda2.in‚3.625% memory_percent‚30.864739418029785 disk.md0p1.out‚ 299.65234375 disk.sdb.out‚310.3955078125 disk.sda2.out‚310.3955078125 proc.cpu.threads‚17( disk./.usagePercent‚83.1681484516645 disk.sda1.in‚0* proc.memory.percent‚18.582844734191895 disk.md0.out‚ 299.65234375 disk.rbd0.out‚ 0.55859375 disk.sdb1.in‚0 disk.sda.out‚310.3955078125 disk.sda1.out‚0 network.sent‚ 4.876284e+06$ disk./.usageGB‚2974.495834350586 disk.md0p1.in‚ 7.12109375 disk.sdb2.in‚ 3.49609375 disk.rbd0.in‚0.015625 disk.sda.in‚3.625" disk.nvme0n1.out‚ 12954.2734375 disk.dm-0.out‚37243.75390625 disk.md0.in‚ 7.12109375 disk.rbd2.in‚0 disk.rbd2.out‚ 0.55859375# disk.nvme1n1.out‚24289.48046875* proc.memory.availableMB‚11327.12109375# proc.memory.rssMB‚ 3044.61328125" disk.nvme1n1.in‚200167.3515625 disk.sdb.in‚ 3.49609375# disk.nvme0n1.in‚328227.71484375 disk.dm-0.in‚528395.06640625˜Õ`Ô2¼- _wandbruntime‚2878  _runtime‚2878±øè»:*½´T TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6966899712 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 111685012 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188859824 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚273 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 175811072 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚382 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚36 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 4886839617024 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚103221775226368 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚98418030807552 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚309 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 565024768 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚65742512 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚44906518 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚17459378 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚45942498 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3532 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6966899712 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20711 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1300 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚309 $ _timestamp‚1.788466825063725e+09 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚109138002967552 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32004 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚44906902 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 175284736 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚45942500 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚65721803 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚273 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 175811072 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚104251139410432 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 4886688272384 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚93662204309280 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚98418030807552 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚384 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 111664301 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚17459760 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 560288769 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚273 trainer/global_step‚2417 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚65721803 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚103221969040384 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚27447140 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚582 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20709 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6741059449 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚93662221348640 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 171820464 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚309 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20711 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3532 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚273 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 175811072 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚104251139410432 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 4886688272384 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚93662204309280 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚98418030807552 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚384 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 111664301 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚17459760 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 560288769 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚273 trainer/global_step‚2417 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚65721803 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚103221969040384 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚27447140 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚582 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20709 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6741059449 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚93662221348640 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 171820464 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚309 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20711 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3532 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 175811072 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32004 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚93687763958912 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3532 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 560288769 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚582 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20709 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 339145216 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚103252314760704 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚309 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6561988608 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 111709302 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚45957880 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 193814016 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚103252314760704 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚104281501164032 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 4790790677007 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6966899712 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 111709302 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32247 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚109168838581248 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20711 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 175284736 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6741059449 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 7134511104 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1319 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚309 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚45957882 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32004 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20709 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884668262191951e+09 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚65751420 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚384 [HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚98478571675279 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 7134511104 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 565024768 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 111688591 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚65730711 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20711 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚36 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚45957880 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚98448078478848 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 151344640 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32247 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚45957882 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 4887162132480 _step‚196 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚98448078478848 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 _runtime‚2879.404496058½õ¶\J)ÁÄR JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚582 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20709 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 339145216 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚103252314760704 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚309 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6561988608 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 111709302 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚45957880 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 193814016 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚103252314760704 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚104281501164032 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 4790790677007 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6966899712 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 111709302 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32247 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚109168838581248 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20711 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 175284736 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6741059449 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 7134511104 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1319 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚309 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚45957882 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32004 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20709 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884668262191951e+09 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚65751420 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚384 [HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚98478571675279 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 7134511104 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 565024768 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 111688591 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚65730711 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20711 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚36 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚45957880 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚98448078478848 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 151344640 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32247 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚45957882 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 4887162132480 _step‚196 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚98448078478848 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 _runtime‚2879.404496058 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 171820464 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 175811072 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32004 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚93687763958912 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3532 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 560288769ÄôVr–’‚(:‡ Š­çÔ“ÉÖˆ gpu.0.gpu‚47.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚119.091' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.77275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0AŒÅpÂ(:å Š­çÔ±¨óˆ disk.md0.in‚ 7.12109375 disk.sdb.out‚312.19384765625 disk.sdb1.in‚0# disk.nvme0n1.in‚328227.71484375* proc.memory.percent‚18.582844734191895( disk./.usagePercent‚83.1682227940554 disk.md0.out‚ 301.41015625 disk.dm-0.in‚528395.06640625 disk.sda2.in‚3.625$ memory_percent‚30.86683750152588 network.recv‚954365 disk.sdb2.in‚ 3.49609375! disk.sdb2.out‚312.19384765625 disk.rbd2.in‚0* proc.memory.availableMB‚11326.77734375 disk.nvme1n1.out‚ 24292.125 disk.sdb.in‚ 3.49609375" disk.nvme0n1.out‚ 12954.3984375# disk./.usageGB‚2974.49849319458 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.12027388429758 disk.md0p1.in‚ 7.12109375 disk.md0p1.out‚ 301.41015625 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 0.55859375 disk.sda.out‚312.19384765625 disk.dm-0.out‚ 37246.5234375! disk.sda2.out‚312.19384765625 proc.cpu.threads‚17 network.sent‚ 4.880425e+06 disk.sdb1.out‚0 disk.rbd0.out‚ 0.55859375" disk.nvme1n1.in‚200167.3515625 disk.sda.in‚3.625# proc.memory.rssMB‚ 3044.61328125‰ñ–Ÿ2Ä- _wandbruntime‚2894  _runtime‚2894øL6ð¢*ÅÕ¥ 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5164474844932556 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 6 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.975907027721405 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9799498915672302 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9875683188438416 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0011337868636474013 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2538147568702698 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4825704097747803 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.24131107330322266 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 *jet/accuracy_2_of_6‚0.5320513248443604 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4060785174369812 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 epoch‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8094976544380188 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 +particle/sensitivity‚0.3267633318901062 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21542753279209137 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3958333432674408 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.30000001192092896 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1531083881855011 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.22827373445034027 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 jet/accuracy_5_of_7‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.30345189571380615 *loss/val_total_loss‚46.188296986452336 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4577043950557709 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.28333333134651184 particle/accuracy_2_of_7‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9047619700431824 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 +"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.25 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.27982455492019653 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3234204649925232 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.25462010502815247 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 jet/accuracy_4_of_6‚0 jet/accuracy_7_of_7‚0 particle/accuracy_1_of_7‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1001797467470169 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.3083333671092987 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ) Purity/0FRt0SRqqt4FBt/FBt_purity‚0.75 particle/accuracy_4_of_6‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3876086175441742 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.48839789628982544 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.32244008779525757 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5681547522544861 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.38533732295036316 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_6_of_7‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5017409324645996 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.48386481404304504 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.3604982793331146 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9906855821609497 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4188596308231354 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17786234617233276 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.26637929677963257 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5438596606254578 ) Purity/0FRt*SRqqt4FBt/FBt_purity‚0.75 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4126445949077606 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.997418999671936 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.024320300668478012 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 particle/accuracy_4_of_4‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9883321523666382 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.22022195160388947 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.7916666865348816 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.982808530330658 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.3125 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.31829124689102173 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.012938963249325752 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 *jet/accuracy_2_of_4‚0.3587276339530945 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18168136477470398 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2529423236846924 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9066194295883179 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.34527072310447693 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.268377423286438 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2669753134250641 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23062428832054138 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9875883460044861 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2064024806022644 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.48148149251937866 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8815789222717285 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_7‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.550000011920929 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.33798423409461975 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18168136477470398 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0013227513991296291 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.22452791035175323 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.24222221970558167 particle/accuracy_6_of_7‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9865824580192566 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.23264823853969574 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.8906250596046448 jet/accuracy_5_of_6‚0 _runtime‚ 2894.06791188 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9859482049942017 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6286703944206238 ,&Purity/1FRt2SRqqt4FBt/event_proži{Ò:hportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_2_of_2‚0.09700985252857208 6 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.800000011920929 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9901543259620667 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.35654905438423157 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735632538795471 particle/accuracy_1_of_3‚1 (particle/accuracy‚0.8501467108726501 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2857142984867096 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4396776556968689 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.33258283138275146 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9950212240219116 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9753968715667725 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 , Purity/3FRt0SRqqt1FBt/FRt_purity‚0.34375 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0476190485060215 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4448727071285248 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2569444179534912 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.01987328939139843 particle/accuracy_2_of_5‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2569444477558136 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17866699397563934 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4891442656517029 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5457525849342346 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.42960017919540405 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9403993487358093 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2569444477558136 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.28916260600090027 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.41747620701789856 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2026071548461914 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.010758377611637115 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2941441535949707 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_7_of_7‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0 _step‚197 particle/accuracy_2_of_3‚1 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3836309611797333 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_6‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0820772722363472 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_2‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_6‚1 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 % _timestamp‚1.7884668408807356e+09 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0063492064364254475 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4568965435028076 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.01941884309053421 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.01908058673143387 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.485268771648407 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2722221910953522 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5060099959373474 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2597152590751648 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 +particle/specificity‚0.9470056295394897 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6590909361839294 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3975047767162323 6 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.294284850358963 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.3104671239852905 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24456648528575897 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.472222238779068 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2856060564517975 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.40997764468193054 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2638888955116272 particle/accuracy_3_of_6‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.989655613899231 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1807963252067566 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3260185718536377 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2005334347486496 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 6 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.402678519487381 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 'particle/f_score‚0.4049781858921051 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.3125 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.04278620705008507 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.3125 +jet/accuracy_3_of_6‚0.07051282376050949 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2708333432674408 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9825748801231384 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996392130851746 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23816247284412384 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.008602150715887547 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0036748009733855724 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_7‚0.5 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.31806623935699463 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6439394354820251 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2708333432674408 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3482615649700165 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.13319647312164307 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17337431013584137 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.07276785373687744 jet/accuracy_4_of_7‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 particle/accuracy_5_of_6‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.288284569978714 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.31877392530441284 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5654278993606567 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.39814817905426025 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2638888955116272 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.016304822638630867 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4384479522705078 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.38333332538604736 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.3125 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.16921234130859375 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.24341025948524475 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.25 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.43906331062316895 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.023148326203227043 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6111111044883728 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 5validation_average_jet_accuracy‚0.409068763256073 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.6000000238418579 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08164562284946442 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.36206895112991333 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ;"Purity/*FRt*SRqqt3FBt/event_purity‚0.042553190141916275 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9895822405815125 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40390488505363464 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9923393726348877 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 )jet/accuracy_1_of_1‚0.450592041015625 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.45196887850761414 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_3‚0.8288511037826538 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.6000000238418579 jet/accuracy_3_of_7‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16107401251792908 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.23524151742458344 jet/accuracy_1_of_6‚1 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08978332579135895 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.36742421984672546 *jet/accuracy_2_of_5‚0.4179147183895111 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.3571428656578064 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1376854032278061 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3680124878883362 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9987789988517761 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008573509752750397 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.40625 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.01886792480945587 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.016167577356100082 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3619930148124695 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.154482901096344 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.02699054405093193 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.2026989907026291 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.03125 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5006120204925537 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 +jet/accuracy_3_of_5‚0.06333845853805542 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3888888955116272 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3747499883174896 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.15909092128276825 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6176470518112183 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3701550364494324 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 *jet/accuracy_1_of_2‚0.7185019254684448 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.14300410449504852 particle/accuracy_5_of_7‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.36721891164779663 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.30777809023857117 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.40625 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2595209777355194 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3718447983264923 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3855196237564087 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3124999701976776 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_7‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4175894558429718 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.15996168553829193 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21340502798557281 particle/accuracy_3_of_3‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4386228621006012 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.41747620701789856 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.21899722516536713 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8924793004989624 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 6 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.222222238779068 jet/accuracy_6_of_6‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.22298428416252136 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.42500001192092896 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.4166666567325592 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9974503517150879 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.997761070728302 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3959052264690399 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.026580460369586945 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.0374622568488121 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4403247535228729 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.42250823974609375 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3073573112487793 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9896395802497864 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4693143963813782 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9885057210922241 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 particle/accuracy_2_of_2‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9993098378181458 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.988077700138092 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.35219261050224304 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9877197742462158 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3649305999279022 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5577087998390198 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9968835711479187 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4187263250350952 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_5‚1 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3199999928474426 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6954853534698486 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.47066110372543335 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.625 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.47881945967674255 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,jet/accuracy_3_of_4‚0.048210036009550095 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4028445780277252 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.02631578966975212 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4999999701976776 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2331734299659729 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0052083334885537624 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3614661395549774 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 particle/accuracy_3_of_5‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9993098378181458 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 trainer/global_step‚2417 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3776368796825409 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9904794096946716 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2971734404563904 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3926745057106018 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2925926446914673 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9890022277832031 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9811320900917053 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25721028447151184 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3958333432674408 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_2_of_7‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.38404130935668945 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.25511011481285095 +"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.25 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.43674004077911377 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.375 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2096147984266281 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.35372021794319153 -jet/accuracy_4_of_4‚0.0021028355695307255 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9753472805023193 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492ú4Ù¸ÆÚ¥ @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0820772722363472 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_2‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_6‚1 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 % _timestamp‚1.7884668408807356e+09 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0063492064364254475 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4568965435028076 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.01941884309053421 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.01908058673143387 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.485268771648407 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2722221910953522 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5060099959373474 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2597152590751648 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 +particle/specificity‚0.9470056295394897 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6590909361839294 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3975047767162323 6 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.294284850358963 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.3104671239852905 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24456648528575897 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.472222238779068 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2856060564517975 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.40997764468193054 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2638888955116272 particle/accuracy_3_of_6‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.989655613899231 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1807963252067566 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3260185718536377 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2005334347486496 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 6 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.402678519487381 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 'particle/f_score‚0.4049781858921051 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.3125 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.04278620705008507 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.3125 +jet/accuracy_3_of_6‚0.07051282376050949 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2708333432674408 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9825748801231384 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996392130851746 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23816247284412384 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.008602150715887547 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0036748009733855724 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_7‚0.5 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.31806623935699463 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6439394354820251 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2708333432674408 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3482615649700165 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.13319647312164307 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17337431013584137 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.07276785373687744 jet/accuracy_4_of_7‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ("Purity/3FRt*SRqqt1FBt/event_purity‚Ò<Åë){0 particle/accuracy_5_of_6‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.288284569978714 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.31877392530441284 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5654278993606567 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.39814817905426025 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2638888955116272 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.016304822638630867 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4384479522705078 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.38333332538604736 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.3125 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.16921234130859375 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.24341025948524475 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.25 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.43906331062316895 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.023148326203227043 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6111111044883728 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 5validation_average_jet_accuracy‚0.409068763256073 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.6000000238418579 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08164562284946442 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.36206895112991333 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ;"Purity/*FRt*SRqqt3FBt/event_purity‚0.042553190141916275 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9895822405815125 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40390488505363464 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9923393726348877 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 )jet/accuracy_1_of_1‚0.450592041015625 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.45196887850761414 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_3‚0.8288511037826538 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.6000000238418579 jet/accuracy_3_of_7‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16107401251792908 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.23524151742458344 jet/accuracy_1_of_6‚1 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08978332579135895 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.36742421984672546 *jet/accuracy_2_of_5‚0.4179147183895111 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.3571428656578064 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1376854032278061 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3680124878883362 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9987789988517761 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008573509752750397 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.40625 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.01886792480945587 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.016167577356100082 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3619930148124695 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.154482901096344 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.02699054405093193 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.2026989907026291 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.03125 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5006120204925537 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 +jet/accuracy_3_of_5‚0.06333845853805542 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3888888955116272 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3747499883174896 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.15909092128276825 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6176470518112183 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3701550364494324 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 *jet/accuracy_1_of_2‚0.7185019254684448 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.14300410449504852 particle/accuracy_5_of_7‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.36721891164779663 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.30777809023857117 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.40625 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2595209777355194 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3718447983264923 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3855196237564087 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3124999701976776 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_7‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4175894558429718 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.15996168553829193 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21340502798557281 particle/accuracy_3_of_3‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4386228621006012 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.41747620701789856 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.21899722516536713 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8924793004989624 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 6 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.222222238779068 jet/accuracy_6_of_6‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.22298428416252136 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.42500001192092896 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.4166666567325592 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9974503517150879 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.997761070728302 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3959052264690399 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.026580460369586945 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.0374622568488121 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4403247535228729 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.42250823974609375 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3073573112487793 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9896395802497864 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4693143963813782 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9885057210922241 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 particle/accuracy_2_of_2‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9993098378181458 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.988077700138092 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.35219261050224304 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9877197742462158 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3649305999279022 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5577087998390198 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9968835711479187 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4187263250350952 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_5‚1 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3199999928474426 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6954853534698486 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.47066110372543335 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.625 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.47881945967674255 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,jet/accuracy_3_of_4‚0.048210036009550095 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4028445780277252 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.02631578966975212 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4999999701976776 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2331734299659729 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0052083334885537624 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3614661395549774 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 particle/accuracy_3_of_5‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9993098378181458 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 trainer/global_step‚2417 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3776368796825409 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9904794096946716 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2971734404563904 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3926745057106018 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2925926446914673 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9890022277832031 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9811320900917053 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25721028447151184 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3958333432674408 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_2_of_7‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.38404130935668945 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.25511011481285095 +"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.25 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.43674004077911377 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.375 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2096147984266281 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.35372021794319153 -jet/accuracy_4_of_4‚0.0021028355695307255 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9753472805023193 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5164474844932556 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 6 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.975907027721405 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9799498915672302 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9875683188438416 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0011337868636474013 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2538147568702698 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4825704097747803 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.24131107330322266 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 *jet/accuracy_2_of_6‚0.5320513248443604 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4060785174369812 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 epoch‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8094976544380188 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 +particle/sensitivity‚0.3267633318901062 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21542753279209137 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3958333432674408 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.30000001192092896 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1531083881855011 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.22827373445034027 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 jet/accuracy_5_of_7‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.30345189571380615 *loss/val_total_loss‚46.188296986452336 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4577043950557709 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.28333333134651184 particle/accuracy_2_of_7‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9047619700431824 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 +"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.25 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.27982455492019653 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3234204649925232 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.25462010502815247 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 jet/accuracy_4_of_6‚0 jet/accuracy_7_of_7‚0 particle/accuracy_1_of_7‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1001797467470169 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.3083333671092987 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ) Purity/0FRt0SRqqt4FBt/FBt_purity‚0.75 particle/accuracy_4_of_6‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3876086175441742 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.48839789628982544 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.32244008779525757 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5681547522544861 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.38533732295036316 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_6_of_7‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5017409324645996 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.48386481404304504 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.3604982793331146 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9906855821609497 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4188596308231354 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17786234617233276 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.26637929677963257 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5438596606254578 ) Purity/0FRt*SRqqt4FBt/FBt_purity‚0.75 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4126445949077606 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.997418999671936 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.024320300668478012 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 particle/accuracy_4_of_4‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9883321523666382 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.22022195160388947 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.7916666865348816 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.982808530330658 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.3125 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.31829124689102173 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.012938963249325752 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 *jet/accuracy_2_of_4‚0.3587276339530945 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18168136477470398 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2529423236846924 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9066194295883179 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.34527072310447693 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.268377423286438 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2669753134250641 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23062428832054138 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9875883460044861 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2064024806022644 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.48148149251937866 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8815789222717285 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_7‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.550000011920929 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.33798423409461975 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18168136477470398 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0013227513991296291 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.22452791035175323 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.24222221970558167 particle/accuracy_6_of_7‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9865824580192566 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.23264823853969574 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.8906250596046448 jet/accuracy_5_of_6‚0 _runtime‚ 2894.06791188 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9859482049942017 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6286703944206238 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_2_of_2‚0.09700985252857208 6 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.800000011920929 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9901543259620667 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.35654905438423157 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735632538795471 particle/accuracy_1_of_3‚1 (particle/accuracy‚0.8501467108726501 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2857142984867096 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4396776556968689 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.33258283138275146 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9950212240219116 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9753968715667725 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 , Purity/3FRt0SRqqt1FBt/FRt_purity‚0.34375 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0476190485060215 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4448727071285248 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2569444179534912 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.01987328939139843 particle/accuracy_2_of_5‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2569444477558136 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17866699397563934 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4891442656517029 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5457525849342346 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.42960017919540405 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9403993487358093 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2569444477558136 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.28916260600090027 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.41747620701789856 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2026071548461914 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.010758377611637115 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2941441535949707 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_7_of_7‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0 _step‚197 particle/accuracy_2_of_3‚1 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3836309611797333 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_6‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5Å" Qí‘Ç‚(:† ™­çÔʲ®ˆ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dŸXh1È‚(:æ ™­çÔ‰½‰ disk.md0.in‚ 7.12109375" proc.memory.rssMB‚ 3109.2578125 cpu‚25.14048619666583 disk.rbd0.in‚0.015625" disk.nvme1n1.in‚200167.3515625# disk.nvme1n1.out‚24520.32421875 disk.sda1.in‚0! disk.sda2.out‚313.87841796875! disk.sdb2.out‚313.87841796875 disk.md0.out‚ 303.0234375 disk.md0p1.out‚ 303.0234375 disk.sdb.in‚ 3.49609375 disk.sdb1.in‚0 disk.sda.in‚3.625% memory_percent‚31.257152557373047) proc.memory.percent‚18.97740364074707 proc.cpu.threads‚17( disk./.usagePercent‚83.1421617862789 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.sda.out‚313.87841796875# disk.nvme0n1.out‚12954.44140625 disk.sda2.in‚3.625( proc.memory.availableMB‚ 11262.828125 disk.sdb2.in‚ 3.49609375 disk.rbd0.out‚ 0.zì_2@55859375 disk.rbd2.out‚ 0.55859375 disk.sdb.out‚313.87841796875# disk.nvme0n1.in‚328227.71484375 disk.dm-0.in‚528395.06640625 disk.sda1.out‚0 disk.md0p1.in‚ 7.12109375 network.sent‚ 4.98719e+06 network.recv‚960148 disk.dm-0.out‚ 37474.765625% disk./.usageGB‚2973.5664253234863[×vÏ’É‚(:‡ ¨­çԳ︇ gpu.0.gpu‚14.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚144.489' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.12225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N¼ÚrÊ‚(:ç ¨­çÔõÒ·$ disk./.usageGB‚2973.858180999756 disk.sdb.out‚315.22705078125! disk.sdb2.out‚315.22705078125 disk.rbd0.out‚ 0.55859375 disk.sda.in‚3.625# disk.nvme0n1.in‚328227.71484375% memory_percent‚31.954026222229004* proc.memory.percent‚18.625569343566895) disk./.usagePercent‚83.15031939709665 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 0.55859375" disk.nvme1n1.in‚200167.3515625" disk.nvme1n1.out‚ 24949.4765625 disk.md0.out‚304.3125 cpu‚25.139030943336323 network.sent‚ 5.110131e+06 disk.rbd2.in‚0 disk.sdb.in‚ 3.49609375 disk.dm-0.in‚528395.06640625 disk.sda1.in‚0 disk.md0p1.out‚304.3125* proc.memory.availableMB‚11148.65234375 disk.sdb1.out‚0# disk.nvme0n1.out‚12954.56640625 disk.md0.in‚ 7.12109375 proc.cpu.threads‚17 disk.sda2.in‚3.625 disk.sdb1.in‚0 disk.sda1.out‚0 disk.sdb2.in‚ 3.49609375# proc.memory.rssMB‚ 3051.61328125 network.recv‚965656 disk.sda.out‚315.22705078125 disk.dm-0.out‚37904.04296875! disk.sda2.out‚315.22705078125 disk.md0p1.in‚ 7.12109375HNwŒ‘Ë‚(:† ·­çÔ¬›Ô… gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚138.642' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.6605 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 113398119 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚104928036998656 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 113424464 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 trainer/global_step‚2449 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚95183497523632 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4871471276032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚667 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17747800 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚104928036998656 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4955580890112 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66700091 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45694019 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45693350 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚100040906282123 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚104928288669696 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27945552 % _timestamp‚1.7884668753913224e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚100056799390720 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17748467 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚95183480484272 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4857425797851 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66673748 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4871471276032 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚669 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248Š6Lßß'ÒÙO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17748467 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚95183480484272 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4857425797851 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66673748 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4871471276032 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚669 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 113398119 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚104928036998656 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 113424464 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 trainer/global_step‚2449 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚95183497523632 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4871471276032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚667 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17747800 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚104928036998656 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4955580890112 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66700091 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45694019 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45693350 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚100040906282123 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚104928288669696 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27945552 % _timestamp‚1.7884668753913224e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚100056799390720 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0ǶÊQ~2Ó- _wandbruntime‚2929  _runtime‚2929:†¯î| ÔÝM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46742648 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17754567 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308255232 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚100080641149181 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66726809 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚95221010841696 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 113443086 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45711558 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113469457 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚110967693017600 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46742648 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66700440 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚100096103430656 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚104969287061504 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4873435315200 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4957293760000 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚200 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2449 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 _runtime‚ 2929.56007545 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45710919 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113469457 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚106010091002368 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4859384286837 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66726809 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4873201633792 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚110967384762368 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4873201633792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27956352 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚58çâg2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚104969538745856 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚100096085427712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚100096103430656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46742646 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66726809 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚95221010841696 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 113443086 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45711558 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113469457 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚110967693017600 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46742648 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66700440 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚100096103430656 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚104969287061504 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4873435315200 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4957293760000 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚200 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2449 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 _runtime‚ 2929.56007545 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45710919 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113469457 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚106010091002368 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4859384286837 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66726809 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4873201633792 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚110967384762368 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4873201633792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27956352 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚104969538745856È*Sú¡2Ö- _wandbruntime‚2929  _runtime‚2929Û‘Y×Ó 1loss/SRqqt1/assignment_loss‚1.560683724284172 /loss/FRt3/detection_loss‚1.4582853188427787 % _timestamp‚1.7884668763768563e+09 _runtime‚2929.561673727 /loss/FRt2/assignment_loss‚3.491701556576623 0loss/FRt3/assignment_loss‚1.5982958765493498 1loss/SRqqt2/assignment_loss‚1.634446848753012 2loss/SRqqt4/assignment_loss‚1.9226169706352296 /loss/FBt3/detection_loss‚0.7045209671987195 /loss/FBt4/detection_loss‚2.7164122263590493 0loss/FBt1/assignment_loss‚0.1153052429144689 1loss/FBt3/assignment_loss‚0.12832048399926876 0loss/FRt1/assignment_loss‚3.5068595837801695 /loss/FRt4/assignment_loss‚3.209043411108164 .loss/FRt4/detection_loss‚3.082982886296052 1loss/SRqqt4/detection_loss‚2.6030104083399617 .loss/FBt2/detection_loss‚2.524392399522993 2loss/SRqqt3/assignment_loss‚1.2461412176093523 0loss/SRqqt1/detection_loss‚2.431960349423545 0loss/SRqqt3/detection_loss‚0.895977262447172 _step‚201 0loss/SRqqt2/detection_loss‚2.196658897731039 %loss/total_loss‚45.52127138495512 2loss/FBt2/assignment_loss‚0.004525469370289809 1loss/FBt4/assignment_loss‚0.18615156895183738 .loss/FRt1/detection_loss‚2.876100143417716 /loss/FRt2/detection_loss‚2.2738741629614703 epoch‚2 trainer/global_step‚2449 /loss/FBt1/detection_loss‚3.1530044078826904˜jcÚ^ØØ 1loss/SRqqt1/assignment_loss‚1.560683724284172 /loss/FRt3/detection_loss‚1.4582853188427787 % _timestamp‚1.7884668763768563e+09 _runtime‚2929.561673727 /loss/FRt2/assignment_loss‚3.491701556576623 0loss/FRt3/assignment_loss‚1.5982958765493498 1loss/SRqqt2/assignment_loss‚1.634446848753012 2loss/SRqqt4/assignment_loss‚1.9226169706352296 /loss/FBt3/detection_loss‚0.7045209671987195 /loss/FBt4/detection_loss‚2.7164122263590493 0loss/FBt1/assignment_loss‚0.1153052429144689 1loss/FBt3/assignment_loss‚0.12832048399926876 0loss/FRt1/assignment_loss‚3.5068595837801695 /loss/FRt4/assignment_loss‚3.209043411108164 .loss/FRt4/detection_loss‚3.082982886296052 1loss/SRqqt4/detection_loss‚2.6030104083399617 .loss/FBt2/detection_loss‚2.524392399522993 2loss/SRqqt3/assignment_loss‚1.2461412176093523 0loss/SRqqt1/detection_loss‚2.431960349423545 0loss/SRqqt3/detection_loss‚0.895977262447172 _step‚201 0loss/SRqqt2/detection_loss‚2.196658897731039 %loss/total_loss‚45.52127138495512 2loss/FBt2/assignment_loss‚0.004525469370289809 1loss/FBt4/assignment_loss‚0.18615156895183738 .loss/FRt1/detection_loss‚2.876100143417716 /loss/FRt2/detection_loss‚2.2738741629614703 epoch‚2 trainer/global_step‚2449 /loss/FBt1/detection_loss‚3.1530044078826904ÉV"ú’Ù‚(:‡ Æ­çÔÞш gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚139.251' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.81275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µu–sÚ‚(:è Æ­çÔ«òŸ‰ disk.sda.in‚3.625 cpu‚25.1378514572413 disk.dm-0.in‚528395.06640625 disk.dm-0.out‚ 38194.046875 disk.sdb2.in‚ 3.49609375 disk.md0.out‚ 307.5703125* proc.memory.availableMB‚11148.89453125$ disk./.usageGB‚2973.991653442383 disk.sdb1.out‚0# proc.memory.rssMB‚ 3051.61328125 network.sent‚ 5.121704e+06 network.recv‚973807" disk.nvme1n1.out‚ 25239.2265625 disk.sdb.in‚ 3.49609375! disk.sda2.out‚318.58935546875 disk.md0p1.in‚ 7.12109375 disk.rbd0.in‚0.015625# disk.nvme0n1.in‚328227.71484375" disk.nvme0n1.out‚ 12954.8203125 disk.sda2.in‚3.625! disk.sdb2.out‚318.58935546875 disk.md0.in‚ 7.12109375 disk.rbd0.out‚ 0.55859375 disk.rbd2.in‚0 disk.rbd2.out‚ 0.55859375" disk.nvme1n1.in‚200167.3515625 disk.sda1.out‚0 disk.md0p1.out‚ 307.5703125% memory_percent‚31.952548027038574* proc.memory.percent‚18.625569343566895 proc.cpu.threads‚17 disk.sdb.out‚318.58935546875 disk.sda.out‚318.58935546875 disk.sda1.in‚0) disk./.usagePercent‚83.15405134245506 disk.sdb1.in‚0IÉŠÛ‚(:… Õ­çÔÎçï‡ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚139.528' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pl9yÜ‚(:î Õ­çÔë𬈠disk.sda.in‚3.625 disk.rbd0.out‚ 0.55859375% memory_percent‚31.960153579711914) proc.memory.availableMB‚ 11147.6484375* proc.memory.percent‚18.625569343566895! disk.sda2.out‚320.18017578125 disk.md0p1.in‚ 7.12109375 disk.dm-0.out‚38311.48046875" disk.nvme1n1.in‚200167.3515625 disk.sdb.out‚320.18017578125 proc.cpu.threads‚17 network.sent‚ 5.231199e+06 network.recv‚978942) disk./.usagePercent‚83.15405176909718$ disk./.usageGB‚2973.991668701172 disk.sdb1.in‚0 disk.sdb2.in‚ 3.49609375 disk.md0.in‚ 7.12109375 disk.rbd2.in‚0 disk.sda1.in‚0! disk.sdb2.out‚320.18017578125 disk.rbd2.out‚ 0.55859375# disk.nvme1n1.out‚25356.61328125# proc.memory.rssMB‚ 3051.61328125 disk.sda1.out‚0 disk.md0p1.out‚ 309.09765625 disk.rbd0.in‚0.015625 disk.sdb.in‚ 3.49609375 cpu‚25.137175293078023 disk.sdb1.out‚0# disk.nvme0n1.in‚328227.71484375" disk.nvme0n1.out‚ 12954.8671875 disk.sda2.in‚3.625 disk.md0.out‚ 309.09765625 disk.sda.out‚320.18017578125 disk.dm-0.in‚528395.06640625N|>pŽÝ‚(:ƒ ä­çԋǼ‡ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚90.236' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚22.559 gpu.0.smClock‚1170 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Q½hiÞ‚(:Þ ä­çԴ network.sent‚ 5.234827e+06) disk./.usagePercent‚83.15875379199524 disk.rbd0.out‚ 0.55859375# disk.nvme1n1.out‚25632.01171875# proc.memory.rssMB‚ 3051.61328125! disk.sda2.out‚322.11376953125" disk.nvme1n1.in‚200167.3515625 disk.sdb.in‚ 3.49609375 disk.sdb.out‚322.11376953125$ memory_percent‚31.21051788330078 disk.dm-0.out‚38587.01171875 disk.sdb2.in‚ 3.49609375 disk.md0.out‚ 310.96875' proc.memory.availableMB‚ 11270.46875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda.in‚3.625# disk.nvme0n1.in‚328227.71484375 disk.dm-0.in‚528395.06640625 disk.sda1.in‚0 disk.md0p1.out‚ 310.96875* proc.memory.percent‚18.625569343566895% disk./.usageGB‚2974.1598358154297 disk.sda.out‚322.11376953125 disk.nvme0n1.out‚12955 disk.sda2.in‚3.625 disk.md0p1.in‚ 7.12109375 disk.rbd0.in‚0.015625 cpu‚25.136489343652425 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd2.in‚0 network.recv‚982658 disk.rbd2.out‚ 0.55859375 disk.md0.in‚ 7.12109375! disk.sdb2.out‚322.11376953125V³<2ß- _wandbruntime‚2982  _runtime‚2982\ž;ÆàÀ & lr-AdamW/pg1‚0.0007961895230442277 & lr-AdamW/pg2‚0.0007961895230442277 trainer/global_step‚2499 $ _timestamp‚1.788466929809209e+09 _runtime‚2982.994147845 _step‚202F³ËáÅ & lr-AdamW/pg1‚0.0007961895230442277 & lr-AdamW/pg2‚0.0007961895230442277 trainer/global_step‚2499 $ _timestamp‚1.788466929809209e+09 _runtime‚2982.994147845 _step‚202Ê%Á2â- _wandbruntime‚2982  _runtime‚2982~N€Ù'ãÓO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚106990554554880 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚102015362193831 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚106990806225920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18093103 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47638121 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68011700 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚106990554554880 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚734 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28485650 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 115676166 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4955592819439 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582Bxß~ äØO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4969686982656 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚113108085766144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚102021101240320 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚106990554554880 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚734 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28485650 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 115676166 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4955592819439 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚106990554554880 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚102015362193831 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚106990806225920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18093103 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47638121 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68011700 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚113151756957696 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4971651021824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28496454 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 115721159 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚102055097060889 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47656398 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚108094313444864 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚97097545752256 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚108094289504768 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚102054851040241 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28496452 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4957322327345 _runtime‚2983.942506569 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚102060387277312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 115694788 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚107032056302080 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4971417340416 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68038392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚204 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47656398 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68064761 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46597545 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5057157100544 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68064761 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46596885 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚658 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68038392 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115721159 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352384 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18100433 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273>¶_é&çãM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚658 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68038392 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115721159 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352384 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18100433 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚107031804617728 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚113151756957696 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4971651021824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28496454 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 115721159 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚102055097060889 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47656398 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚108094313444864 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚97097545752256 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚108094289504768 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚102054851040241 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28496452 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4957322327345 _runtime‚2983.942506569 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚102060387277312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 115694788 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚107032056302080 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4971417340416 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68038392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚204 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47656398 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68064761 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46597545 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5057157100544 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68064761 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46596885 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0Ì»‡k2è- _wandbruntime‚2983  _runtime‚2983è‚ôZéÔ 0loss/FBt1/assignment_loss‚0.3312804505985696 /loss/FRt3/detection_loss‚1.4626467456260035 1loss/SRqqt1/detection_loss‚1.8580067132910092 /loss/FBt4/detection_loss‚2.4446048378944396 trainer/global_step‚2499 2loss/SRqqt1/assignment_loss‚1.6795381502558788 /loss/FRt2/detection_loss‚2.0263848914680156 .loss/FBt1/detection_loss‚4.334628269076347 2loss/SRqqt3/assignment_loss‚1.0670359604198232 .loss/FRt4/detection_loss‚2.914864929392934 &loss/total_loss‚46.627293893611636 % _timestamp‚1.7884669307587507e+09 0loss/FBt4/assignment_loss‚0.4770502802333795 1loss/SRqqt2/detection_loss‚2.2552437947856054 1loss/SRqqt4/detection_loss‚2.5464148661669563 epoch‚2 _runtime‚2983.943494834 0loss/FRt1/assignment_loss‚2.5556436700698657 0loss/FRt3/assignment_loss‚1.8625217214707406 /loss/FRt4/assignment_loss‚4.083244381472468 1loss/SRqqt3/detection_loss‚0.9077315239913084 1loss/SRqqt4/assignment_loss‚1.907650508424815 .loss/FBt2/detection_loss‚2.303431270023187 .loss/FBt3/detection_loss‚0.719728303395889 2loss/SRqqt2/assignment_loss‚1.9980430425041251 1loss/FBt3/assignment_loss‚0.12120387099850617 _step‚205 0loss/FRt2/assignment_loss‚3.7093355991623618 /loss/FBt2/assignment_loss‚0.618941455948459 /loss/FRt1/detection_loss‚2.4421186569409494®&N_êÙ epoch‚2 _runtime‚2983.943494834 0loss/FRt1/assignment_loss‚2.5556436700698657 0loss/FRt3/assignment_loss‚1.8625217214707406 /loss/FRt4/assignment_loss‚4.083244381472468 1loss/SRqqt3/detection_loss‚0.9077315239913084 1loss/SRqqt4/assignment_loss‚1.907650508424815 .loss/FBt2/detection_loss‚2.303431270023187 .loss/FBt3/detection_loss‚0.719728303395889 2loss/SRqqt2/assignment_loss‚1.9980430425041251 1loss/FBt3/assignment_loss‚0.12120387099850617 _step‚205 0loss/FRt2/assignment_loss‚3.7093355991623618 /loss/FBt2/assignment_loss‚0.618941455948459 /loss/FRt1/detection_loss‚2.4421186569409494 0loss/FBt1/assignment_loss‚0.3312804505985696 /loss/FRt3/detection_loss‚1.4626467456260035 1loss/SRqqt1/detection_loss‚1.8580067132910092 /loss/FBt4/detection_loss‚2.4446048378944396 trainer/global_step‚2499 2loss/SRqqt1/assignment_loss‚1.6795381502558788 /loss/FRt2/detection_loss‚2.0263848914680156 .loss/FBt1/detection_loss‚4.334628269076347 2loss/SRqqt3/assignment_loss‚1.0670359604198232 .loss/FRt4/detection_loss‚2.914864929392934 &loss/total_loss‚46.627293893611636 % _timestamp‚1.7884669307587507e+09 0loss/FBt4/assignment_loss‚0.4770502802333795 1loss/SRqqt2/detection_loss‚2.2552437947856054 1loss/SRqqt4/detection_loss‚2.5464148661669563Í Ýͨœë‚(:‘ ó­çÔœž‡ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚118.474' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.618500000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™Aøoì‚(:ä ó­çÔŸ÷߈# disk.nvme0n1.out‚12955.06640625 disk.dm-0.out‚ 38794.296875 network.sent‚ 5.25223e+06 disk.rbd0.out‚ 0.55859375# disk.nvme1n1.out‚25839.23046875* proc.memory.availableMB‚10976.29296875 disk.rbd2.out‚ 0.55859375" disk.nvme1n1.in‚200167.3515625$ memory_percent‚33.00602436065674 network.recv‚986377 disk.md0.out‚ 312.390625 disk.sda.out‚323.5986328125 disk.sda2.out‚323.5986328125* proc.memory.percent‚18.625569343566895 disk.md0.in‚ 7.12109375 disk.rbd0.in‚0.015625 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0p1.in‚ 7.12109375# proc.memory.rssMB‚ 3051.61328125 disk.sda.in‚3.625 disk.sda2.in‚3.625) disk./.usagePercent‚83.15889490387923$ disk./.usageGB‚2974.164882659912 disk.rbd2.in‚0 disk.sdb.out‚323.5986328125 disk.sdb1.in‚0 cpu‚25.135805050907308 disk.sdb2.in‚ 3.49609375 disk.sdb.in‚ 3.49609375 proc.cpu.threads‚17 disk.md0p1.out‚ 312.390625 disk.sdb2.out‚323.5986328125# disk.nvme0n1.in‚328227.71484375 disk.dm-0.in‚528395.06640625Y¾úeí‚(:ë ‚®çÔŸÀ‡ disk.rbd2.out‚ 0.55859375" disk.nvme1n1.in‚200167.3515625 disk.sdb.out‚324.97509765625( disk./.usagePercent‚83.1624015821938 disk.sda1.out‚0 disk.sdb.in‚ 3.49609375 disk.sda2.in‚3.625 disk.md0p1.in‚ 7.12109375 disk.md0.in‚ 7.12109375 disk.sda.in‚3.625$ memory_percent‚33.00518989562988* proc.memory.percent‚18.625569343566895 network.recv‚991906 disk.sda1.in‚0! disk.sda2.out‚324.97509765625 disk.rbd0.in‚0.015625 network.sent‚ 5.349149e+06$ disk./.usageGB‚2974.290298461914# disk.nvme0n1.out‚12955.19921875 disk.sda.out‚324.97509765625) proc.memory.availableMB‚ 10976.4296875# proc.memory.rssMB‚ 3051.61328125 cpu‚25.13523757804305 proc.cpu.threads‚17 disk.dm-0.in‚528395.06640625 disk.md0p1.out‚ 313.69921875! disk.sdb2.out‚324.97509765625 disk.rbd2.in‚0 disk.md0.out‚ 313.69921875 disk.rbd0.out‚ 0.55859375 disk.dm-0.out‚38917.50390625" disk.nvme1n1.out‚ 25962.3046875 disk.sdb1.in‚0 disk.sdb1.out‚0# disk.nvme0n1.in‚328227.71484375 disk.sdb€«Å—2.in‚ 3.49609375iÿœî‚(:‘ ‚®çÔìäé‡ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚110.064' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚27.515999999999995 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OGànï‚(:ã ‘®çÔ瀪‡* proc.memory.percent‚18.625569343566895 network.recv‚996459 disk.sdb.out‚326.3681640625 disk.sda.in‚3.625 disk.sda1.out‚0 network.sent‚ 5.35275e+06 disk.sda2.out‚326.3681640625 disk.sdb2.out‚326.3681640625% memory_percent‚33.008384704589844 cpu‚25.13457510607051 disk.dm-0.in‚528395.06640625 disk.md0p1.out‚ 315.05078125 proc.cpu.threads‚17! disk.nvme1n1.out‚ 26040.796875) disk./.usagePercent‚83.16591028705848$ disk./.usageGB‚2974.415786743164 disk.sda1.in‚0' proc.memory.availableMB‚ 10975.90625# proc.memory.rssMB‚ 3051.61328125 disk.rbd2.in‚0 disk.sdb2.in‚ 3.49609375 disk.md0.in‚ 7.12109375 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 0.55859375 disk.sdb1.out‚0# disk.nvme0n1.in‚328227.71484375 disk.rbd0.out‚ 0.55859375 disk.sdb.in‚ 3.49609375 disk.sdb1.in‚0 disk.sda.out‚326.3681640625" disk.nvme0n1.out‚ 12955.2578125 disk.sda2.in‚3.625 disk.md0p1.in‚ 7.12109375 disk.md0.out‚ 315.05078125" disk.nvme1n1.in‚200167.3515625 disk.dm-0.out‚ 38996.0546875«f‰R’ð‚(:‡ ‘®çÔìŸé‡ gpu.0.gpu‚45.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚136.743' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.18575 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Fkxñ‚(:í  ®çÔО“‡ disk.sda.out‚328.4267578125 disk.sda1.in‚0# disk.nvme0n1.in‚328227.71484375# disk.nvme0n1.out‚12955.31640625 disk.sda1.out‚0 disk.md0p1.out‚ 317.05859375% memory_percent‚33.006906509399414# proc.memory.rssMB‚ 3051.61328125* proc.memory.percent‚18.625569343566895 disk.sdb2.out‚328.4267578125 disk.nvme1n1.out‚ 26445.4375 disk.sdb.in‚ 3.49609375 disk.sda2.in‚3.625 network.recv‚ 1.000519e+06) disk./.usagePercent‚83.17288620584084 disk.rbd0.in‚0.015625 disk.sdb1.in‚0 disk.dm-0.out‚39400.75390625 disk.sdb2.in‚ 3.49609375% disk./.usageGB‚2974.6652793884277 disk.rbd2.in‚0" disk.nvme1n1.in‚200167.3515625 cpu‚25.13408263945066 disk.md0.out‚ 317.05859375 disk.md0p1.in‚ 7.12109375) proc.memory.availableMB‚ 10976.1484375 proc.cpu.threads‚17 network.sent‚ 5.358844e+06 disk.md0.in‚ 7.12109375 disk.dm-0.in‚528395.06640625 disk.sda2.out‚328.4267578125 disk.rbd0.out‚ 0.55859375 disk.rbd2.out‚ 0.55859375 disk.sdb.out‚328.4267578125 disk.sdb1.out‚0 disk.sda.in‚3.625L‚|‘ò‚(:†  ®çÔÙµŸ‡ gpu.0.gpu‚34.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚135.802' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.9505 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãw-72ó- _wandbruntime‚3040  _runtime‚3040ÚiÎèÇôÁ & lr-AdamW/pg2‚0.0007939840316794854 trainer/global_step‚2549 % _timestamp‚1.7884669874807875e+09 _runtime‚3040.665818412 _step‚206 & lr-AdamW/pg1‚0.0007939840316794854@|ÆÌõÆ & lr-AdamW/pg1‚0.0007939840316794854 & lr-AdamW/pg2‚0.0007939840316794854 trainer/global_step‚2549 % _timestamp‚1.7884669874807875e+09 _runtime‚3040.665818412 _step‚206Îî†è2ö- _wandbruntime‚3040  _runtime‚30402§iÙ'÷ÓO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚115291910420992 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5155106383872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69375995 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚109053323782144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5067669021184 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18439668 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5053530859947 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 trainer/global_step‚2549 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29025750 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5067669021184 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48551873 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48551871 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚115292222883840 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5067902689280 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48551873 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5155394906624 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚692 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚109053072111104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 117901523 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚694 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚110136804037120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47466112 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚110136827977216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 117927868 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69375995 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚103985421092864 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚103985403089920 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚98936533266032 % _timestamp‚1.7884669874823382e+09 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚109053072111104 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚103990064125979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚692 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚109053072111104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 117901523 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚694 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚110136804037120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47466112 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚110136827977216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 117927868 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69375995 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚103985421092864 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚103985403089920 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚98936533266032 % _timestamp‚1.7884669874823382e+09 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚109053072111104 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚103990064125979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48551873 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48551871 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚115292222883840 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5067902689280 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48551873 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453ÏJIkA2ù- _wandbruntime‚3041  _runtime‚3041”ª×w úÞM D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48570148 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚109094573858304 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚104029552972597 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117972861 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48570148 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _step‚208 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 117946490 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚711 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884669886484256e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48570146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740992 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚109094322173952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18446828 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚98974063623456 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69402713 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69402713 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚104024689126912 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚713 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47483380 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5157098617856 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚98974046584096 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69376344 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚104024707129856 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 ;5DeviceStatsMonitorS#2Ü.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5069866728448 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29036552 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 117972861 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 117946490 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69376344 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5055489349141 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288'ÎtWé&ûãM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚109094573858304 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚104029552972597 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117972861 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48570148 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _step‚208 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 117946490 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚711 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884669886484256e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48570146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740992 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚109094322173952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18446828 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚98974063623456 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69402713 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69402713 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚104024689126912 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚713 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47483380 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5157098617856 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚98974046584096 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69376344 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚104024707129856 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5069866728448 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29036552 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 117972861 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 117946490 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69376344 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5055489349141 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48570148 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582ÐÖÐÀž2ü- _wandbruntime‚3041  _runtime‚3041bå÷~VýÐ %loss/total_loss‚43.90374980782502 0loss/FRt4/assignment_loss‚3.3843114683122346 2loss/SRqqt1/assignment_loss‚1.2671151176715891 0loss/FBt2/assignment_loss‚0.2315984909366509 0loss/FBt4/assignment_loss‚0.1834482006314649 % _timestamp‚1.7884669886517184e+09 0loss/FRt3/assignment_loss‚1.4850617756306286 /loss/FBt3/detection_loss‚0.4892860834069293 2loss/SRqqt4/assignment_loss‚1.7867157166557652 /loss/FRt3/detection_loss‚1.5331153652932443 1loss/SRqqt3/detection_loss‚0.7531399207676316 /loss/FBt1/detection_loss‚2.3993443571604214 trainer/global_step‚2549 0loss/FRt2/assignment_loss‚3.7410925797053745 1loss/SRqqt2/assignment_loss‚1.825126616480319 1loss/FBt1/assignment_loss‚0.04643456367416722 .loss/FRt1/detection_loss‚2.695785756046708 0loss/SRqqt4/detection_loss‚2.256384883608137 _step‚209 /loss/FRt1/assignment_loss‚3.282304029206972 epoch‚2 1loss/SRqqt1/detection_loss‚2.1998924960692725 _runtime‚3041.836487316 0loss/SRqqt3/assignment_loss‚1.12208376601674 0loss/FBt3/assignment_loss‚0.1555200468625844 -loss/FRt2/detection_loss‚2.18869016833681 /loss/FRt4/detection_loss‚2.9707354198802602 0loss/SRqqt2/detection_loss‚2.485050740994905 .loss/FBt2/detection_loss‚2.992661522494422 /loss/FBt4/detection_loss‚2.4288507219817905Â%ñö[þÕ trainer/global_step‚2549 0loss/FRt2/assignment_loss‚3.7410925797053745 1loss/SRqqt2/assignment_loss‚1.825126616480319 1loss/FBt1/assignment_loss‚0.04643456367416722 .loss/FRt1/detection_loss‚2.695785756046708 0loss/SRqqt4/detection_loss‚2.256384883608137 _step‚209 /loss/FRt1/assignment_loss‚3.282304029206972 epoch‚2 1loss/SRqqt1/detection_loss‚2.1998924960692725 _runtime‚3041.836487316 0loss/SRqqt3/assignment_loss‚1.12208376601674 0loss/FBt3/assignment_loss‚0.1555200468625844 -loss/FRt2/detection_loss‚2.18869016833681 /loss/FRt4/detection_loss‚2.9707354198802602 0loss/SRqqt2/detection_loss‚2.485050740994905 .loss/FBt2/detection_loss‚2.992661522494422 /loss/FBt4/detection_loss‚2.4288507219817905 %loss/total_loss‚43.90374980782502 0loss/FRt4/assignment_loss‚3.3843114683122346 2loss/SRqqt1/assignment_loss‚1.2671151176715891 0loss/FBt2/assignment_loss‚0.2315984909366509 0loss/FBt4/assignment_loss‚0.1834482006314649 % _timestamp‚1.7884669886517184e+09 0loss/FRt3/assignment_loss‚1.4850617756306286 /loss/FBt3/detection_loss‚0.4892860834069293 2loss/SRqqt4/assignment_loss‚1.7867157166557652 /loss/FRt3/detection_loss‚1.5331153652932443 1loss/SRqqt3/detection_loss‚0.7531399207676316 /loss/FBt1/detection_loss‚2.3993443571604214ÑÁ¸ Žÿ‚(:ƒ ¯®çÔü°¶‡ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚137.36' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.34 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚ºŸ|€‚(:ñ ¯®çÔ°Ùˆ disk.md0.out‚ 318.7578125 disk.dm-0.in‚528395.06640625 disk.sda1.in‚0! disk.sda2.out‚330.19287109375 proc.cpu.threads‚17 disk.sdb.in‚ 3.49609375) disk./.usagePercent‚83.17209126489493 disk.md0.in‚ 7.12109375 disk.rbd0.out‚ 0.55859375" disk.nvme1n1.in‚200167.3515625 disk.sda.out‚330.19287109375 disk.rbd2.out‚ 0.55859375# disk.nvme0n1.in‚328227.71484375! disk.sdb2.out‚330.19287109375$ memory_percent‚33.00907611846924 network.sent‚ 5.37711e+06 network.recv‚ 1.004208e+06* proc.memory.availableMB‚10975.79296875$ disk./.usageGB‚2974.636848449707 disk.rbd2.in‚0 disk.sda.in‚3.625" disk.nvme0n1.out‚ 12955.4609375 disk.sdb2.in‚ 3.49609375# proc.memory.rssMB‚ 3051.61328125 cpu‚25.133565023415493 disk.sdb.out‚330.19287109375 disk.rbd0.in‚0.015625 disk.sdb1.out‚0 disk.dm-0.out‚39649.89453125 disk.sda2.in‚3.625 disk.md0p1.in‚ 7.12109375 disk.md0p1.out‚ 318.7578125* proc.memory.percent‚18.625569343566895# disk.nvme1n1.out‚26694.43359375 disk.sdb1.in‚0 disk.sda1.out‚0Ñúœœ‚(:‘ ¾®çÔ‰‚Úˆ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Sëîm‚‚(:â ¾®çÔ—úõˆ) disk./.usagePercent‚83.17067587963368 disk.sdb1.in‚0 disk.sda2.in‚3.625 disk.sdb2.out‚ 331.6328125 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.md0p1.out‚ 320.12890625# proc.memory.rssMB‚ 3051.61328125 network.recv‚ 1.009838e+06# disk.nvme1n1.out‚26695.17578125# disk.nvme0n1.in‚328227.71484375 disk.sda1.out‚0 disk.sda2.out‚ 331.6328125 disk.md0.out‚ 320.12890625$ memory_percent‚33.00774097442627* proc.memory.availableMB‚10976.01171875 disk.sdb.out‚ 331.6328125 disk.dm-0.in‚528395.06640625 disk.rbd0.in‚0.015625 disk.dm-0.out‚39650.80859375 disk.sdb2.in‚ 3.49609375* proc.memory.percent‚18.625569343566895 disk.md0.in‚ 7.12109375 network.sent‚ 5.471871e+06 disk.sda.in‚3.625" disk.nvme0n1.out‚ 12955.6328125 disk.sda1.in‚0 disk.md0p1.in‚ 7.12109375 cpu‚25.1329943323624 proc.cpu.threads‚17$ disk./.usageGB‚2974.586227416992 disk.sda.out‚ 331.6328125 disk.rbd0.out‚ 0.55859375 disk.rbd2.out‚ 0.55859375" disk.nvme1n1.in‚200167.3515625 disk.sdb.in‚ 3.49609375Kd*1’ƒ‚(:‡ Í®çÔÝÒÔ‡ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚126.829' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.70725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uLg}„‚(:ò Í®çÔú¹µˆ disk.md0.out‚ 321.9453125# disk.nvme1n1.out‚26895.15234375 disk.sda.in‚3.625# disk.nvme0n1.in‚328227.71484375! disk.sda2.out‚333.50634765625 disk.md0p1.out‚ 321.9453125 network.recv‚ 1.013305e+06 disk.sdb.out‚333.50634765625* proc.memory.availableMB‚10976.36328125$ disk./.usageGB‚2974.670757293701 disk.sdb.in‚ 3.49609375% memory_percent‚33.005595207214355# proc.memory.rssMB‚ 3051.61328125 cpu‚25.132521916394605 proc.cpu.threads‚17 disk.md0.in‚ 7.12109375 disk.rbd2.out‚ 0.63671875 disk.sda1.in‚0 disk.sdb2.in‚ 3.49609375! disk.sdb2.out‚333.50634765625 disk.rbd0.in‚0.015625* proc.memory.percent‚18.625569343566895 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.sda.out‚333.50634765625 disk.sda1.out‚0 disk.sda2.in‚3.625 disk.md0p1.in‚ 7.12109375 network.sent‚ 5.479723e+06 disk.dm-0.in‚528395.06640625 disk.dm-0.out‚39931.84765625) disk./.usagePercent‚83.17303937036496 disk.rbd0.out‚ 0.5703125" disk.nvme1n1.in‚200167.3515625 disk.sdb1.in‚0" disk.nvme0n1.out‚ 13036.6953125úR å‘…‚(:† Ü®çÔ‡¯‡ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚129.614' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.4035 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îCNo†‚(:ä Ü®çÔ´ÿ™ˆ disk.sda2.in‚3.625! disk.sda2.out‚335.11767578125 disk.md0p1.out‚323.5 disk.sdb2.in‚ 3.49609375 disk.md0.in‚ 7.12109375# disk.nvme0n1.in‚328227.71484375 disk.dm-0.in‚528395.06640625 disk.rbd0.out‚ 0.6328125# disk.nvme0n1.out‚13068.51171875 disk.sdb1.out‚0( proc.memory.availableMB‚ 10976.359375$ disk./.usageGB‚2974.735813140869! disk.sdb2.out‚335.11767578125 disk.rbd2.in‚0 disk.sdb.out‚335.11767578125) disk./.usagePercent‚83.17485835907976 disk.rbd2.out‚ 0.63671875 network.sent‚ 5.482818e+06 disk.dm-0.out‚ 40096.7109375 disk.md0p1.in‚ 7.12109375 disk.rbd0.in‚0.015625# disk.nvme1n1.out‚27028.19921875 disk.sda.out‚335.11767578125 cpu‚25.132049232849866 network.recv‚ 1.017404e+06 disk.sda1.out‚0" disk.nvme1n1.in‚200167.3515625 disk.sdb.in‚ 3.49609375 disk.sdb1.in‚0 disk.sda.in‚3.625# proc.memory.rssMB‚ 3051.61328125* proc.memory.percent‚18.625569343566895 disk.sda1.in‚0 disk.md0.out‚323.5% memory_percent‚33.005619049072266 proc.cpu.threads‚17gC®Ò2‡- _wandbruntime‚3098  _runtime‚3098¨–HLjÁ & lr-AdamW/pg1‚0.0007916989825862965 & lr-AdamW/pg2‚0.0007916989825862965 trainer/global_step‚2599 % _timestamp‚1.7884670454949045e+09 _runtime‚3098.679828036 _step‚210~€ÖŒÌ‰Æ % _timestamp‚1.7884670454949045e+09 _runtime‚3098.679828036 _step‚210 & lr-AdamW/pg1‚0.0007916989825862965 & lr-AdamW/pg2‚0.0007916989825862965 trainer/global_step‚2599Òå·â¬2Š- _wandbruntime‚3098  _runtime‚3098¢ðšÜ'‹ÖO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48352898 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29565852 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18787728 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70713947 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49465623 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 120179570 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70713947 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚111115589667328 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚112221002539520 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5166118395904 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18787048 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5255346897920 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚105964520037687 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚105949722942464 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚105949722942464 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70687604 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚117476060914688 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5151468900455 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚100813051137232 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 trainer/global_step‚2599 _runtime‚3098.681110832 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29565850 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70687604 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49465623 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚111115841338368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5151239919375 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚105964274017247 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 _step‚211 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5166118395904 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5165884727808 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49465621 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 120153225 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚117476373377536 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112221026479616 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚105949704939520 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 % _timestamp‚1.7884670454959736e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5165884727808 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49465621 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309L\2ÌÞŒÛO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49465623 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 120179570 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 XIDeviceStatsMonitor.on_train_batch_start/reser^í×~#ved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70713947 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚111115589667328 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚112221002539520 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5166118395904 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18787048 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5255346897920 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚105964520037687 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚105949722942464 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚105949722942464 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70687604 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚117476060914688 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5151468900455 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚100813051137232 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 trainer/global_step‚2599 _runtime‚3098.681110832 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29565850 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70687604 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49465623 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚111115841338368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5151239919375 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚105964274017247 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 _step‚211 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5166118395904 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5165884727808 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49465621 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 120153225 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚117476373377536 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112221026479616 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚105949704939520 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 % _timestamp‚1.7884670454959736e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5165884727808 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49465621 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48352898 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29565852 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18787728 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70713947 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343ÓÎ}è¤2- _wandbruntime‚3099  _runtime‚3099¤¶‰æ&ŽàM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5153427389649 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚112262686509568 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5167848753664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚669 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚105988990976512 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5168082435072 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48371607 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49483898 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚117520051157504 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚105989008979456 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚111157091414528 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49483896 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48370938 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29576654 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 120198192 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49483896 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚106003762863657 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5153198408361 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70714296 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120224563 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 120198192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29576652 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5257340707840 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚106004008884305 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5168082435072 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚111156839730176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105989008979456 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 trainer/global_step‚2599 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5257064781312 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚100850564455296 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275926528 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _runtime‚3099.621918587 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚667 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚117519751290880 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5167848753664 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7884670464363704e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408‰×¿‰ë&åM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5257340707840 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚106004008884305 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5168082435072 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚111156839730176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105989008979456 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 trainer/global_step‚2599 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5257064781312 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚100850564455296 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275926528 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _runtime‚3099.621918587 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚667 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚117519751290880 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5167848753664 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7884670464363704e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5153427389649 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚112262686509568 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5167848753664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚669 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚105988990976512 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5168082435072 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48371607 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49483898 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚117520051157504 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚105989008979456 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚111157091414528 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49483896 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48370938 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29576654 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 120198192 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49483896 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚106003762863657 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5153198408361 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70714296 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120224563 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 120198192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29576652Ô1UB2- _wandbruntime‚3099  _runtime‚3099¸-ZÐV‘Ð 1loss/SRqqt3/assignment_loss‚1.306922524049878 .loss/FRt1/detection_loss‚3.188253933475131 .loss/FRt2/detection_loss‚2.121671046011357 .loss/FBt2/detection_loss‚2.592617710431417 0loss/FBt3/detection_loss‚0.42908989285287036 /loss/FRt4/assignment_loss‚3.094209600578655 1loss/SRqqt4/assignment_loss‚1.751671895823058 0loss/FBt3/assignment_loss‚0.0657185072437197 0loss/SRqqt4/detection_loss‚2.445186607977923 trainer/global_step‚2599 0loss/FRt3/assignment_loss‚1.5065959662687582 2loss/SRqqt1/assignment_loss‚1.7859165791284155 1loss/FBt1/assignment_loss‚0.29221713952154005 0loss/SRqqt3/detection_loss‚1.042883685686522 _runtime‚3099.623776515 /loss/FRt2/assignment_loss‚3.745283305644989 1loss/FBt4/assignment_loss‚0.13452936848610383 .loss/FRt3/detection_loss‚1.687620786369824 /loss/FBt1/detection_loss‚2.2105166713396707 % _timestamp‚1.7884670464390872e+09 1loss/FBt2/assignment_loss‚0.24724253156455234 0loss/FRt1/assignment_loss‚3.3810376553308394 .loss/FRt4/detection_loss‚3.087199319492687 1loss/SRqqt2/detection_loss‚3.0384146802955203 /loss/FBt4/detection_loss‚2.2577341198921204 epoch‚2 1loss/SRqqt2/assignment_loss‚1.963006133834521 1loss/SRqqt1/detection_loss‚1.9654268953535292 $loss/total_loss‚45.3409665566536 _step‚213 c,…[’Õ 1loss/SRqqt1/detection_loss‚1.9654268953535292 $loss/total_loss‚45.3409665566536 _step‚213 1loss/SRqqt3/assignment_loss‚1.306922524049878 .loss/FRt1/detection_loss‚3.188253933475131 .loss/FRt2/detection_loss‚2.121671046011357 .loss/FBt2/detection_loss‚2.592617710431417 0loss/FBt3/detection_loss‚0.42908989285287036 /loss/FRt4/assignment_loss‚3.094209600578655 1loss/SRqqt4/assignment_loss‚1.751671895823058 0loss/FBt3/assignment_loss‚0.0657185072437197 0loss/SRqqt4/detection_loss‚2.445186607977923 trainer/global_step‚2599 0loss/FRt3/assignment_loss‚1.5065959662687582 2loss/SRqqt1/assignment_loss‚1.7859165791284155 1loss/FBt1/assignment_loss‚0.29221713952154005 0loss/SRqqt3/detection_loss‚1.042883685686522 _runtime‚3099.623776515 /loss/FRt2/assignment_loss‚3.745283305644989 1loss/FBt4/assignment_loss‚0.13452936848610383 .loss/FRt3/detection_loss‚1.687620786369824 /loss/FBt1/detection_loss‚2.2105166713396707 % _timestamp‚1.7884670464390872e+09 1loss/FBt2/assignment_loss‚0.24724253156455234 0loss/FRt1/assignment_loss‚3.3810376553308394 .loss/FRt4/detection_loss‚3.087199319492687 1loss/SRqqt2/detection_loss‚3.0384146802955203 /loss/FBt4/detection_loss‚2.2577341198921204 epoch‚2 1loss/SRqqt2/assignment_loss‚1.963006133834521Õ%gÙ8“‚(:… ë®çÔàÕψ gpu.0.gpu‚37.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚103.544' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.886 gpu.0.smClock‚1215 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÇžhjH”‚(:ð ë®çÔŸïÁ‰$ disk./.usageGB‚2974.923667907715 disk.md0.in‚ 7.12109375 disk.md0.out‚ 324.9453125 disk.rbd0.out‚ 0.6328125 disk.sda.in‚3.625 disk.dm-0.out‚40355.19140625 disk.md0p1.in‚ 7.12109375# proc.memory.rssMB‚ 3051.61328125* proc.memory.percent‚18.625569343566895 cpu‚25.13155544498344 network.sent‚ 5.488093e+06 network.recv‚ 1.021264e+06! disk.nvme1n1.out‚ 27227.890625 disk.sdb.in‚ 3.49609375 disk.sdb.out‚336.62158203125 disk.rbd2.out‚ 0.72412109375 disk.sdb1.in‚0 disk.sdb1.out‚0# diÊA&—3sk.nvme0n1.in‚328227.71484375 disk.sda1.in‚0 proc.cpu.threads‚17( disk./.usagePercent‚83.1801108569841! disk.sdb2.out‚336.62158203125 disk.md0p1.out‚ 324.9453125 disk.sda.out‚336.62158203125# disk.nvme0n1.out‚13127.30078125 disk.dm-0.in‚528395.06640625! disk.sda2.out‚336.62158203125$ memory_percent‚33.00576210021973 disk.sdb2.in‚ 3.49609375 disk.rbd0.in‚0.015625 disk.rbd2.in‚0" disk.nvme1n1.in‚200167.3515625 disk.sda1.out‚0 disk.sda2.in‚3.625) proc.memory.availableMB‚ 10976.3359375î!C’•‚(:‡ ú®çÔ“Þׇ gpu.0.gpu‚46.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚126.829' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.70725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³åéi€–‚(:õ ú®çÔññ‡ disk.rbd0.out‚ 0.7119140625 disk.sdb.out‚338.02392578125 disk.sdb1.out‚0 disk.sda2.in‚3.625$ memory_percent‚33.00752639770508( proc.memory.availableMB‚ 10976.046875# proc.memory.rssMB‚ 3051.61328125 proc.cpu.threads‚17 network.sent‚ 5.595758e+06) disk./.usagePercent‚83.18711504080747 disk.md0.in‚ 7.12109375# disk.nvme1n1.out‚27427.41796875 disk.sda1.in‚0 disk.sdb2.in‚ 3.49609375" disk.nvme1n1.in‚200167.3515625 disk.sdb1.in‚0 disk.dm-0.in‚528395.06640625 disk.rbd0.in‚0.015625 disk.sda.in‚3.625 disk.sda1.out‚0* proc.memory.percent‚18.625569343566895 cpu‚25.131036130597362 disk.md0.out‚ 326.2890625 disk.rbd2.in‚0 disk.md0p1.in‚ 7.12109375 disk.md0p1.out‚ 326.2890625 disk.sda.out‚338.02392578125# disk.nvme0n1.in‚328227.71484375 disk.dm-0.out‚40701.26953125 network.recv‚ 1.027528e+06 disk.sdb.in‚ 3.49609375 disk.rbd2.out‚ 0.72412109375$ disk./.usageGB‚2975.174171447754! disk.sdb2.out‚338.02392578125" disk.nvme0n1.out‚ 13273.8515625! disk.sda2.out‚338.02392578125@UP•’—‚(:‡ ‰¯çÔÙš…ˆ gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚129.947' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.48675 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñüâ´{˜‚(:𠉯çÔØûìˆ" disk.nvme1n1.in‚200167.3515625! disk.nvme1n1.out‚ 27427.828125 disk.sdb1.out‚0# disk.nvme0n1.out‚13391.32421875 disk.sda2.in‚3.625 disk.md0.out‚ 328.05859375 disk.rbd2.in‚0 disk.sda1.out‚0$ memory_percent‚33.00604820251465) proc.memory.availableMB‚ 10976.2890625# proc.memory.rssMB‚ 3051.61328125 disk.dm-0.in‚528395.06640625 proc.cpu.threads‚17$ disk./.usageGB‚2975.165340423584 disk.rbd2.out‚ 0.787109375# disk.nvme0n1.in‚328227.71484375 disk.dm-0.out‚40819.15234375 disk.sdb1.in‚0 disk.md0.in‚ 7.12109375 disk.sda1.in‚0 network.recv‚ 1.031955e+06( disk./.usagePercent‚83.1868681216756 disk.rbd0.in‚0.015625 disk.sdb.in‚ 3.49609375 disk.sdb.out‚339.8505859375 disk.sda.in‚3.625 disk.sda2.out‚339.8505859375 disk.md0p1.out‚ 328.05859375 disk.sdb2.in‚ 3.49609375 disk.sda.out‚339.8505859375 disk.md0p1.in‚ 7.12109375* proc.memory.percent‚18.625569343566895 cpu‚25.130557545198297 network.sent‚ 5.603889e+06 disk.sdb2.out‚339.8505859375 disk.rbd0.out‚ 0.7119140625 ­=û™‚(:… ˜¯çÔÏî͇ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚116.916' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.229 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´mIš‚(:ô ˜¯çÔáÕƒˆ disk.sdb1.in‚0 disk.sdb1.out‚0# disk.nvme0n1.out‚13537.87890625 disk.sda1.out‚0 disk.md0p1.out‚ 329.73828125 disk.sdb2.in‚ 3.49609375 disk.sdb.in‚ 3.49609375* proc.memory.percent‚18.625569343566895 network.recv‚ 1.036094e+06 disk.sda.out‚341.59423828125# disk.nvme0n1.in‚328227.71484375! disk.sda2.out‚341.59423828125 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 0.77490234375# proc.memory.rssMB‚ 3051.61328125 network.sent‚ 5.60773e+06$ disk./.usageGB‚2975.165340423584 disk.dm-0.out‚ 41165.21875 disk.sda2.in‚3.625 disk.md0.in‚ 7.12109375$ memory_percent‚33.00602436065674" disk.nvme1n1.in‚200167.3515625 proc.cpu.threads‚17 disk.sda.in‚3.625 disk.dm-0.in‚528395.06640625 disk.md0p1.in‚ 7.12109375 disk.md0.out‚ 329.73828125( disk./.usagePercent‚83.1868681216756 disk.sda1.in‚0! disk.sdb2.out‚341.59423828125# disk.nvme1n1.out‚27627.33984375* proc.memory.availableMB‚10976.29296875 cpu‚25.130112903834178 disk.rbd2.in‚0 disk.rbd2.out‚ 0.787109375 disk.sdb.out‚341.59423828125­r±þ2›- _wandbruntime‚3155  _runtime‚3155ÂbÇœÁ & lr-AdamW/pg2‚0.0007893348519654672 trainer/global_step‚2649 % _timestamp‚1.7884671027826128e+09 _runtime‚3155.967544795 _step‚214 & lr-AdamW/pg1‚0.0007893348519654672>U&ËÌÆ & lr-AdamW/pg1‚0.0007893348519654672 & lr-AdamW/pg2‚0.0007893348519654672 trainer/global_step‚2649 % _timestamp‚1.7884671027826128e+09 _runtime‚3155.967544795 _step‚214Ö1ë!2ž- _wandbruntime‚3155  _runtime‚3155XµF»Û'ŸÕO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5355301095424 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚102689551969072 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 $ _timestamp‚1.788467102783599e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72025556 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5264334102528 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚107914006789120 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚114305201041920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49239514 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462848 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚692 trainer/global_step‚2649 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚113178107223552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50379371 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5264100434432 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚107914024792064 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5264100434432 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚119660526077440 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚690 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚119660213614592 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚107938975949395 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5249177959883 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50379373 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50379373 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49238822 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚107938729928955 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚3155.968693409 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50379371 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72051899 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122431272 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19132872 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113178107223552 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72051899 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5249406940963 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚114305224982016 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚102689569008432 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19133562 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚113178358894592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30105952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚113178358894592 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50379373 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50379373 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49238822 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚107938729928955 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚3155.968693409 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50379371 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72051899 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122431272 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19132872 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113178107223552 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72051899 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5249406940963 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚114305224982016 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚102689569008432 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19133562 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚113178358894592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30105952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚113178358894592 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49239514 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462848 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚692 trainer/global_step‚2649 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚113178107223552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50379371 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5264100434432 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚107914024792064 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5264100434432 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚119660526077440 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚690 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0×b9†Þ2¡- _wandbruntime‚3156  _runtime‚3156C~NR¢àM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72052248 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49256289 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚119703895880704 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30116752 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122476265 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72078617 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5357295183872 _step‚216 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚113219608970752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚113219608970752 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122449894 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5266298141696 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚107953310829056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚102727082326496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50397646 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚107953292826112 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308255232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30116754 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚107953292826112 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122449894 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 % _timestamp‚1.7884671037511141e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50397648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352icë&£åM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72052248 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49256289 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚119703895880704 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30116752 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122476265 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72078617 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5357295183872 _step‚216 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚113219608970752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚113219608970752 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122449894 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5266298141696 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚107953310829056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚102727082326496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50397646 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚107953292826112 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308255232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30116754 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚107953292826112 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122449894 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 % _timestamp‚1.7884671037511141e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50397648Øþ -2¤- _wandbruntime‚3156  _runtime‚3156ù£Â\¥Ö $ _timestamp‚1.788467103753052e+09 1loss/FBt4/assignment_loss‚0.06635907000401046 /loss/FRt1/detection_loss‚3.0192743301391602 0loss/SRqqt2/detection_loss‚2.122043728022962 /loss/FBt2/detection_loss‚2.3471141116959706 %loss/total_loss‚45.75687701411229 2loss/SRqqt4/assignment_loss‚1.6135730929672718 1loss/FBt3/assignment_loss‚0.07574173351526452 /loss/FRt2/assignment_loss‚3.863764270146688 /loss/FBt4/detection_loss‚2.9681514609943735 epoch‚2 .loss/FRt4/assignment_loss‚3.38259448059674 1loss/FBt2/assignment_loss‚0.16228809912822076 1loss/SRqqt1/detection_loss‚2.1476011018861425 1loss/SRqqt3/detection_loss‚0.8468736950078128 _runtime‚3156.937782461 _step‚217 0loss/FRt3/assignment_loss‚1.4564588879471394 2loss/SRqqt1/assignment_loss‚1.6678345917554742 1loss/FBt1/assignment_loss‚0.13008434600620108 /loss/FRt3/detection_loss‚1.8080917392204057 /loss/FRt2/detection_loss‚2.4370043977137863 .loss/FRt4/detection_loss‚2.776881986650927 0loss/SRqqt4/detection_loss‚2.938240577777227 /loss/FBt1/detection_loss‚2.4015656276182695 trainer/global_step‚2649 0loss/FRt1/assignment_loss‚3.6410287797451018 2loss/SRqqt2/assignment_loss‚1.9548959828711845 2loss/SRqqt3/assignment_loss‚1.0836707100180196 /loss/FBt3/detection_loss‚0.8457402126839408“ÀGa¦Û 2loss/SRqqt3/assignment_loss‚1.0836707100180196 /loss/FBt3/detection_loss‚0.8457402126839408 $ _timestamp‚1.788467103753052e+09 1loss/FBt4/assignment_loss‚0.06635907000401046 /loss/FRt1/detection_loss‚3.0192743301391602 0loss/SRqqt2/detection_loss‚2.122043728022962 /loss/FBt2/detection_loss‚2.3471141116959706 %loss/total_loss‚45.75687701411229 2loss/SRqqt4/assignment_loss‚1.6135730929672718 1loss/FBt3/assignment_loss‚0.07574173351526452 /loss/FRt2/assignment_loss‚3.863764270146688 /loss/FBt4/detection_loss‚2.9681514609943735 epoch‚2 .loss/FRt4/assignment_loss‚3.38259448059674 1loss/FBt2/assignment_loss‚0.16228809912822076 1loss/SRqqt1/detection_loss‚2.1476011018861425 1loss/SRqqt3/detection_loss‚0.8468736950078128 _runtime‚3156.937782461 _step‚217 0loss/FRt3/assignment_loss‚1.4564588879471394 2loss/SRqqt1/assignment_loss‚1.6678345917554742 1loss/FBt1/assignment_loss‚0.13008434600620108 /loss/FRt3/detection_loss‚1.8080917392204057 /loss/FRt2/detection_loss‚2.4370043977137863 .loss/FRt4/detection_loss‚2.776881986650927 0loss/SRqqt4/detection_loss‚2.938240577777227 /loss/FBt1/detection_loss‚2.4015656276182695 trainer/global_step‚2649 0loss/FRt1/assignment_loss‚3.6410287797451018 2loss/SRqqt2/assignment_loss‚1.9548959828711845ÙØâ&J‘§‚(:† §¯çÔŒí¸ˆ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.357' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.83925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0È·Ny¨‚(:î §¯çÔî÷óˆ disk.md0p1.in‚ 7.12109375 disk.md0.out‚ 331.1328125 disk.rbd0.out‚ 0.77490234375 disk.md0p1.out‚ 331.1328125 disk.sdb2.in‚ 3.49609375 disk.sda2.in‚3.625! disk.sda2.out‚343.05517578125" disk.nvme1n1.in‚200167.3515625 disk.sdb.in‚ 3.49609375 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.sda.in‚3.625 network.sent‚ 5.625817e+06 disk.md0.in‚ 7.12109375 disk.sdb.out‚343.05517578125 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.nvme1n1.out‚ 27760.46875 disk.sdb1.out‚0 disk.nvme0n1.out‚ 13610.09375 disk.sda1.out‚0) proc.memory.availableMB‚ 10976.2734375 disk.rbd2.out‚ 0.85009765625 network.recv‚ 1.039647e+06$ memory_percent‚33.00614356994629* proc.memory.percent‚18.625569343566895! disk.sdb2.out‚343.05517578125# disk.nvme0n1.in‚328227.71484375 disk.dm-0.out‚ 41370.5625# proc.memory.rssMB‚ 3051.61328125 cpu‚25.129644580236175 disk.sda.out‚343.05517578125 disk.dm-0.in‚528395.06640625 disk.sda1.in‚0( disk./.usagePercent‚83.1868687616388% disk./.usageGB‚2975.1653633117676È5L˜‘©‚(:† ¶¯çÔÒ뻇 gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚104.494' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚26.1235 gpu.0.smClock‚1215 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x}zª‚(:ô ¶¯çÔŽÏ©ˆ* proc.memory.availableMB‚10976.27734375 disk.sdb.in‚ 3.49609375 disk.dm-0.in‚528395.06640625 disk.md0p1.in‚ 7.12109375 disk.md0p1.out‚ 333.1328125 disk.rbd2.in‚0" disk.nvme1n1.in‚200167.3515625$ memory_percent‚33.00611972808838# disk.nvme0n1.in‚328227.71484375 disk.sda1.in‚0 disk.sda2.out‚345.1220703125 disk.sdb.out‚345.1220703125 cpu‚25.129049056394233 proc.cpu.threads‚17 disk.dm-0.out‚ 41624.828125 disk.sda1.out‚0# disk.nvme1n1.out‚27893.49609375 disk.sda.in‚3.625# disk.nvme0n1.out‚13731.33203125 disk.md0.in‚ 7.12109375 disk.md0.out‚ 333.1328125 network.recv‚ 1.045468e+06% disk./.usageGB‚2975.1653442382812 disk.sda.out‚345.1220703125 disk.rbd0.in‚0.015625 disk.sdb1.in‚0 disk.sdb1.out‚0# proc.memory.rssMB‚ 3051.61328125* proc.memory.percent‚18.625569343566895) disk./.usagePercent‚83.18686822833614 disk.sda2.in‚3.625 disk.rbd2.out‚ 0.85009765625 network.sent‚ 5.720694e+06 disk.sdb2.in‚ 3.49609375 disk.sdb2.out‚345.1220703125 disk.rbd0.out‚ 0.77490234375) % ’«‚(:‡ ůçÔøÚ¹‡ gpu.0.gpu‚42.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚133.017' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.25425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ð*„¬‚(:ö ůçÔ•ó͇ disk.dm-0.in‚528395.06640625 disk.sda1.out‚0! disk.sdb2.out‚346.39892578125$ disk./.usageGB‚2975.165454864502 disk.md0p1.in‚ 7.12109375 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 0.85009765625" disk.nvme1n1.in‚200167.3515625 disk.sda.in‚3.625# disk.nvme0n1.in‚328227.71484375 disk.sda1.in‚0! disk.sda2.out‚346.39892578125 disk.rbd0.out‚ 0.837890625 disk.rbd2.in‚0# disk.nvme1n1.out‚28093.12109375# disk.nvme0n1.out‚13801.93359375 disk.sda2.in‚3.625 disk.sdb1.out‚0# proc.memory.rssMB‚ 3051.61328125 cpu‚25.128549277908796 disk.sdb.out‚346.39892578125 disk.sda.out‚346.39892578125 disk.dm-0.out‚ 41895.0546875 disk.sdb.in‚ 3.49609375 network.sent‚ 5.728866e+06 network.recv‚ 1.049913e+06) disk./.usagePercent‚83.18687132149158 disk.md0p1.out‚ 334.3515625 disk.md0.out‚ 334.3515625 disk.sdb1.in‚0$ memory_percent‚33.00344944000244* proc.memory.percent‚18.625569343566895 disk.sdb2.in‚ 3.49609375 disk.md0.in‚ 7.12109375* proc.memory.availableMB‚10976.71484375 proc.cpu.threads‚17*€Ê›­‚(: Ô¯çÔÁ©‡ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚117.81' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.452499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õŽÍßx®‚(:í Ô¯çÔš”ᇠcpu‚25.128018624946897 network.recv‚ 1.053869e+06! disk.nvme1n1.out‚ 28205.171875 disk.sdb1.out‚0 proc.cpu.threads‚17 disk.sda1.out‚0 network.sent‚ 5.734884e+06 disk.md0.in‚ 7.12109375# proc.memory.rssMB‚ 3051.61328125 disk.sdb2.in‚ 3.49609375 disk.sdb2.out‚ 347.771484375 disk.rbd2.out‚ 0.85009765625# disk.nvme0n1.in‚328227.71484375* proc.memory.availableMB‚10976.95703125) disk./.usagePercent‚83.15729019617346 disk.rbd2.in‚0 disk.sdb.out‚ 347.771484375 disk.sda.out‚ 347.771484375 disk.dm-0.in‚528395.06640625* proc.memory.percent‚18.625569343566895$ disk./.usageGB‚2974.107490539551 disk.md0.out‚ 335.6640625 disk.rbd0.in‚0.015625 disk.sdb.in‚ 3.49609375 disk.sda.in‚3.625 disk.sda2.in‚3.625 disk.md0p1.in‚ 7.12109375 disk.md0p1.out‚ 335.6640625 disk.rbd0.out‚ 0.837890625" disk.nvme1n1.in‚200167.3515625 disk.sdb1.in‚0# disk.nvme0n1.out‚14003.19140625 disk.dm-0.out‚42208.36328125 disk.sda1.in‚0 disk.sda2.out‚ 347.771484375$ memory_percent‚33.00197124481201ÒP¸Ê2¯- _wandbruntime‚3210  _runtime‚3210SwðIÅ°¿ _runtime‚3210.543204606 _step‚218 % lr-AdamW/pg1‚0.000786892132498275 % lr-AdamW/pg2‚0.000786892132498275 trainer/global_step‚2699 % _timestamp‚1.7884671573583028e+09ÃŒµc汀 % lr-AdamW/pg1‚0.000786892132498275 % lr-AdamW/pg2‚0.000786892132498275 trainer/global_step‚2699 % _timestamp‚1.7884671573583028e+09 _runtime‚3210.543204606 _step‚218Ú-xK,2²- _wandbruntime‚3210  _runtime‚3210Ìþó$á%³ÖO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚121844693568000 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50122685 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚109913431861103 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 124656629 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚115240624779776 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124682974 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51293121 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚716 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30646050 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116389423484416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5347116000391 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5362549809152 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚109878308638720 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 _step‚219 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚104566069840272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚121844381105152 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19476635 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚2699 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50123401 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚116389399544320 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 % _timestamp‚1.7884671573593953e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51293123 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚115240624779776 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51293123 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚714 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚115240876450816 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚109878326641664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5362549809152 UGDeviceStatsMoniô›ûtor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5455270083584 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚109878326641664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2ìÖõÔá'´ÛO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5347344981471 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚115240876450816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30646052 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19477349 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5362316141056 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚104566086879632 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73389851 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚109878308638720 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462848 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚115240624779776 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124682974 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51293121 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚716 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30646050 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116389423484416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5347116000391 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5362549809152 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚109878308638720 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 _step‚219 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚104566069840272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚121844381105152 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19476635 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚2699 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50123401 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚116389399544320 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 % _timestamp‚1.7884671573593953e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51293123 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚115240624779776 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51293123 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚714 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚115240876450816 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚109878326641664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5362549809152 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5455270083584 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚109878326641664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚121844693568000 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50122685 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚109913431861103 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 124656629 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124727967 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚115282126526976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚116431107454464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5457260133376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 124701596 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5364280166912 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5349303470665 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73390200 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚109917594675712 # _timestamp‚1.78846715828971e+09 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚109952674687073 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚109917594675712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5364513848320 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚116431083514368 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚109917612678656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121888367587840 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 trainer/global_step‚2699 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50140187 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚104603617237056 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352384 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚220 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73390200 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30656852 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚121888057235456 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51311396 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚3211.475257228 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30656854 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5349074489377 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚115281874842624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19484009 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124727967 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚115282126526976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚116431107454464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5457260133376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 124701596 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5364280166912 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5349303470665 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73390200 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚109917594675712 # _timestamp‚1.78846715828971e+09 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚109952674687073 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚109917594675712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5364513848320 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚116431083514368 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚109917612678656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121888367587840 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 trainer/global_step‚2699 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50140187 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚104603617237056 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352384 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚220 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73390200 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30656852 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚121888057235456 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51311396 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚3211.475257228 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30656854 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5349074489377 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚115281874842624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19484009 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74701460 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74727803 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31186150 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5445054040899 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚111842610488320 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚118473598046720 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚106442587711472 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5445283021979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚687 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚117303142336000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51010726 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19824576 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚111842628491264 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52206873 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 126908331 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚106442604750832 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52206873 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5460531847680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚685 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚124028832327168 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74701460 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74727803 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31186150 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5445054040899 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚111842610488320 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚118473598046720 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚106442587711472 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5445283021979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚687 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚117303142336000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51010726 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19824576 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚111842628491264 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52206873 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 126908331 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚106442604750832 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52206873 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5460531847680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚685 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚124028832327168 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚124072215516160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788467209986897e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51028783 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚118515305956864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚2749 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚111881896525312 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚111927376619429 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31196954 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚111881896525312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 _runtime‚3263.172480936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚111881914528256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚111927130598781 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5447012529885 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74754521 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19831831 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5447241511173 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚124072530062848 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚117344392398848 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31196952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 126979669 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19832513 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546688 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 126953298 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚111881914528256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5462729554944 XFDeviccTÕleStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5462495873536 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚118515282016768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚106480118068896 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 126979669 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19832513 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546688 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 126953298 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚111881914528256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5462729554944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5462495873536 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚118515282016768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚106480118068896 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚124072215516160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788467209986897e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51028783 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚118515305956864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚2749 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚111881896525312 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚111927376619429 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31196954 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚111881896525312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 _runtime‚3263.172480936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚111881914528256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚111927130598781 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5447012529885 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74754521 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19831831 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5447241511173 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚124072530062848 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚117344392398848 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31196952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309à-wà2Ê- _wandbruntime‚3263  _runtime‚3263Èë™ÚZËÔ $ _timestamp‚1.788467209989525e+09 _runtime‚3263.174225584 0loss/SRqqt2/detection_loss‚2.115523624114501 0loss/SRqqt4/detection_loss‚2.636867743951303 epoch‚2 /loss/FRt2/detection_loss‚2.0112668701580594 0loss/FRt4/assignment_loss‚3.3538545414253518 1loss/SRqqt2/assignment_loss‚2.384641341674022 0loss/FBt4/assignment_loss‚0.4662388070797673 trainer/global_step‚2749 .loss/FRt1/detection_loss‚2.979934178549668 0loss/FRt3/assignment_loss‚1.2451644171225398 2loss/SRqqt4/assignment_loss‚1.5798147155178919 1loss/FBt1/assignment_loss‚0.19333928348405607 /loss/FRt3/detection_loss‚1.3556601257602636 /loss/FBt2/detection_loss‚2.2343721787134805 %loss/total_loss‚46.39810981532557 /loss/FRt4/detection_loss‚2.9951946625003107 /loss/FBt1/detection_loss‚2.6056729833285015 /loss/FBt3/detection_loss‚0.8895531077795322 1loss/FBt2/assignment_loss‚0.48239105108408087 /loss/FRt1/assignment_loss‚3.844418476367819 1loss/SRqqt1/assignment_loss‚2.075602439897401 2loss/SRqqt3/assignment_loss‚1.0553016411050142 0loss/SRqqt1/detection_loss‚2.809324733912945 /loss/FRt2/assignment_loss‚3.706533590141608 1loss/SRqqt3/detection_loss‚0.8939673253922786 2loss/FBt3/assignment_loss‚0.003522639479634276 /loss/FBt4/detection_loss‚2.4799493367855368 _step‚225Ôä°ÿ_ÌÙ 1loss/FBt2/assignment_loss‚0.48239105108408087 /loss/FRt1/assignment_loss‚3.844418476367819 1loss/SRqqt1/assignment_loss‚2.075602439897401 2loss/SRqqt3/assignment_loss‚1.0553016411050142 0loss/SRqqt1/detection_loss‚2.809324733912945 /loss/FRt2/assignment_loss‚3.706533590141608 1loss/SRqqt3/detection_loss‚0.8939673253922786 2loss/FBt3/assignment_loss‚0.003522639479634276 /loss/FBt4/detection_loss‚2.4799493367855368 _step‚225 $ _timestamp‚1.788467209989525e+09 _runtime‚3263.174225584 0loss/SRqqt2/detection_loss‚2.115523624114501 0loss/SRqqt4/detection_loss‚2.636867743951303 epoch‚2 /loss/FRt2/detection_loss‚2.0112668701580594 0loss/FRt4/assignment_loss‚3.3538545414253518 1loss/SRqqt2/assignment_loss‚2.384641341674022 0loss/FBt4/assignment_loss‚0.4662388070797673 trainer/global_step‚2749 .loss/FRt1/detection_loss‚2.979934178549668 0loss/FRt3/assignment_loss‚1.2451644171225398 2loss/SRqqt4/assignment_loss‚1.5798147155178919 1loss/FBt1/assignment_loss‚0.19333928348405607 /loss/FRt3/detection_loss‚1.3556601257602636 /loss/FBt2/detection_loss‚2.2343721787134805 %loss/total_loss‚46.39810981532557 /loss/FRt4/detection_loss‚2.9951946625003107 /loss/FBt1/detection_loss‚2.6056729833285015 /loss/FBt3/detection_loss‚0.8895531077795322áÀí8‘Í‚(:† °çÔÑ¿þ‡ gpu.0.gpu‚26.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.966' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.9915 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m¬çŽ}΂(:ò °çÔ§Ú̈ network.recv‚ 1.07135e+06 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 0.837890625 disk.sdb.out‚353.9755859375 disk.sdb1.in‚0 disk.sda2.in‚3.625 disk.md0p1.in‚ 7.12109375 disk.md0.in‚ 7.12109375 disk.sda.in‚3.625# disk.nvme0n1.in‚328227.71484375 disk.sda1.in‚0 disk.md0.out‚ 341.61328125# memory_percent‚33.0033540725708* proc.memory.availableMB‚10976.73046875 disk.rbd2.in‚0" disk.nvme1n1.in‚200167.3515625# disk.nvme0n1.out‚14203.33984375 disk.sda2.out‚353.9755859375 disk.sdb2.out‚353.9755859375* proc.memory.percent‚18.625569343566895 network.sent‚ 5.856871e+06 disk.sdb2.in‚ 3.49609375# proc.memory.rssMB‚ 3051.61328125 cpu‚25.126134026757313 disk.rbd2.out‚ 0.85009765625 disk.sda.out‚353.9755859375 disk.md0p1.out‚ 341.61328125) disk./.usagePercent‚83.14288441138405$ disk./.usageGB‚2973.592269897461 disk.sdb.in‚ 3.49609375 disk.sdb1.out‚0 disk.dm-0.out‚ 42492.0546875# disk.nvme1n1.out‚28288.71484375 disk.dm-0.in‚528395.06640625 disk.sda1.out‚0 proc.cpu.threads‚17­ŒL)‘Ï‚(:† Ÿ°çÔµò‘‡ gpu.0.gpu‚3.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚107.279' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚26.81975 gpu.0.smClock‚1200 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 †>|Ђ(:ñ Ÿ°çÔ¾°ê‡" disk.nvme1n1.in‚200167.3515625" disk.nvme1n1.out‚ 28463.2734375# disk.nvme0n1.out‚14203.51953125 disk.sda1.out‚0) disk./.usagePercent‚83.14477272944472 disk.rbd2.out‚ 0.85009765625 disk.sdb.in‚ 3.49609375$ memory_percent‚33.00495147705078 disk.sdb.out‚355.93505859375 disk.sda.in‚3.625 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sdb1.out‚0# disk.nvme0n1.in‚328227.71484375! disk.sda2.out‚355.93505859375 cpu‚25.125651457658122 network.recv‚ 1.07616e+06 disk.md0.out‚ 343.515625 disk.rbd0.in‚0.015625 disk.sdb1.in‚0 network.sent‚ 5.965724e+06 disk.rbd0.out‚ 0.837890625 disk.dm-0.in‚528395.06640625 disk.dm-0.out‚42666.79296875 disk.sda2.in‚3.625 disk.md0p1.out‚ 343.515625' proc.memory.availableMB‚ 10976.46875 disk.sda1.in‚0 disk.md0p1.in‚ 7.12109375! disk.sdb2.out‚355.93505859375% disk./.usageGB‚2973.6598052978516 disk.md0.in‚ 7.12109375 disk.sda.out‚355.93505859375 disk.sdb2.in‚ 3.49609375# proc.memory.rssMB‚ 3051.61328125* proc.memory.percent‚18.625569343566895,ÿþ‘Ñ‚(:† ®°çÔ­²¶‡ gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚115.966' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.9915 gpu.0.smClock‚1245 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ç‚ò|Ò‚(:ñ ®°çÔ‡Ÿé‡ disk.rbd0.out‚ 0.837890625# disk.nvme1n1.out‚28551.27734375* proc.memory.percent‚18.625569343566895" disk.nvme1n1.in‚200167.3515625 disk.sdb.out‚357.3408203125 disk.sdb1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sdb.in‚ 3.49609375 disk.sda.in‚3.625 disk.sda.out‚357.3408203125 disk.dm-0.out‚42754.84765625 disk.sda1.in‚0 disk.sda2.in‚3.625 disk.md0p1.in‚ 7.12109375 disk.md0p1.out‚ 344.859375 disk.md0.in‚ 7.12109375 disk.rbd0.in‚0.015625 disk.sdb1.out‚0$ memory_percent‚33.00492763519287* proc.memory.availableMB‚10976.47265625 disk.md0.out‚ 344.859375 disk.rbd2.in‚0 disk.rbd2.out‚ 0.85009765625 disk.dm-0.in‚528395.06640625 cpu‚25.125236189804404 network.sent‚ 5.972056e+06 network.recv‚ 1.080732e+06% disk./.usageGB‚2973.7902030944824 disk.sdb2.in‚ 3.49609375 disk.sdb2.out‚357.3408203125# proc.memory.rssMB‚ 3051.61328125" disk.nvme0n1.out‚ 14203.5703125) disk./.usagePercent‚83.14841870641423# disk.nvme0n1.in‚328227.71484375 disk.sda2.out‚357.3408203125²Siz’Ó‚(:‡ ½°çÔ¿ü¥‡ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚141.427' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.35675 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºem…‚Ô‚(:÷ ½°çÔ›ô؇" disk.nvme1n1.out‚ 28740.9609375 disk.sdb.in‚ 3.49609375 disk.sdb.out‚358.93701171875# disk.nvme0n1.in‚328227.71484375 disk.rbd2.out‚ 0.85009765625( disk./.usagePercent‚83.1589170892699 disk.rbd0.in‚0.015625$ memory_percent‚33.00492763519287 network.recv‚ 1.08492e+06! disk.sdb2.out‚358.93701171875 disk.dm-0.in‚528395.06640625* proc.memory.availableMB‚10976.47265625* proc.memory.percent‚18.625569343566895 proc.cpu.threads‚17 disk.rbd0.out‚ 0.837890625 disk.sdb1.out‚0# disk.nvme0n1.out‚14203.79296875 disk.md0p1.out‚ 346.39453125! disk.sda2.out‚358.93701171875# proc.memory.rssMB‚ 3051.61328125 disk.sda.out‚358.93701171875 network.sent‚ 5.975657e+06 disk.md0p1.in‚ 7.12109375 disk.md0.in‚ 7.12109375 disk.rbd2.in‚0" disk.nvme1n1.in‚200167.3515625 cpu‚25.124826455761916 disk.sda.in‚3.625 disk.sda1.in‚0 disk.sda1.out‚0% disk./.usageGB‚2974.1656761169434 disk.sdb1.in‚0 disk.dm-0.out‚42944.75390625 disk.sdb2.in‚ 3.49609375 disk.md0.out‚ 346.39453125 disk.sda2.in‚3.625-,©V2Õ- _wandbruntime‚3318  _runtime‚3318€Ê.ÅÖ¿ % lr-AdamW/pg1‚0.000781772979532817 % lr-AdamW/pg2‚0.000781772979532817 trainer/global_step‚2799 % _timestamp‚1.7884672650872796e+09 _runtime‚3318.272470184 _step‚226–X XÊ×Ä % _timestamp‚1.7884672650872796e+09 _runtime‚3318.272470184 _step‚226 % lr-AdamW/pg1‚0.000781772979532817 % lr-AdamW/pg2‚0.000781772979532817 trainer/global_step‚2799â¯Øå(2Ø- _wandbruntime‚3318  _runtime‚3318¹'ª2%ÙÕO LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚126212887460352 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 trainer/global_step‚2799 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚119365911563264 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5558747554304 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚113806912337920 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53120623 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 % _timestamp‚1.7884672650883846e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314560000 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚108319122622032 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5542992081407 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5558747554304 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚120557820489216 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 _step‚227 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚126212572900352 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76065755 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20175990 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 129186378 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51902242 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31726252 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚119365659892224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5655066971136 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚113862097664079 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5543221062487 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53120623 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119365659892224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20175291 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚108319105582672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5558981222400 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 _runtime‚ 3318.27373438 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76065755 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚113806912337920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53120621 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚120557796549120 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚701 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚115¯ódÉ3862343684519 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5558981222400 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚113806930340864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5654776351232 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 bË¢à'ÚÚO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚113862343684519 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5558981222400 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚113806930340864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5654776351232 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚126212887460352 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 trainer/global_step‚2799 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚119365911563264 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5558747554304 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚113806912337920 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53120623 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 % _timestamp‚1.7884672650883846e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314560000 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚108319122622032 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5542992081407 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5558747554304 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚120557820489216 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 _step‚227 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚126212572900352 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76065755 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20175990 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 129186378 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51902242 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31726252 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚119365659892224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5655066971136 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚113862097664079 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5543221062487 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53120623 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119365659892224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20175291 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚108319105582672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5558981222400 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 _runtime‚ 3318.27373438 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76065755 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚113806912337920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53120621 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚120557796549120 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚701 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582ã ¤p2Û- _wandbruntime‚3319  _runtime‚3319[/?å&ÜßM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚119407161639424 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129231371 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚126256269928448 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76092473 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚119407161639424 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚679 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20182665 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5656789409280 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53138896 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5545179551681 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5560945261568 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53138898 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚2799 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚119406909955072 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76066104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53138898 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚677 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20181988 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚113901832531137 _runtime‚3319.358078939 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51919040 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _step‚228 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53138896 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 129205000 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76092473 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚126256584475136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280Tj$ê&ÝäM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5560945261568 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53138898 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚2799 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚119406909955072 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76066104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53138898 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚677 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20181988 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚113901832531137 _runtime‚3319.358078939 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51919040 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _step‚228 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53138896 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 129205000 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76092473 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚126256584475136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚119407161639424 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129231371 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚126256269928448 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76092473 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚119407161639424 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚679 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20182665 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5656789409280 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53138896 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5545179551681 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ä—=·¯2Þ- _wandbruntime‚3319  _runtime‚3319{q4ZßÔ 0loss/FRt2/assignment_loss‚3.6606765181519263 0loss/FRt3/assignment_loss‚0.9674500148804461 0loss/FRt4/assignment_loss‚3.0347230125356606 0loss/FBt2/assignment_loss‚0.6614197936225017 .loss/FBt4/detection_loss‚2.511816886564096 &loss/total_loss‚44.428071145549204 trainer/global_step‚2799 0loss/FRt1/assignment_loss‚2.6881260921557746 /loss/FRt4/detection_loss‚2.8627821648562395 0loss/SRqqt3/detection_loss‚0.961080459093577 epoch‚2 2loss/SRqqt4/assignment_loss‚1.5878309574392107 1loss/FBt4/assignment_loss‚0.17210401103602635 2loss/SRqqt2/assignment_loss‚1.9905798316001893 0loss/SRqqt2/detection_loss‚2.674694974081857 .loss/FRt2/detection_loss‚2.550875496804216 _runtime‚3319.359097563 /loss/FBt1/assignment_loss‚0.513518924242544 .loss/FBt2/detection_loss‚3.158429537500654 $ _timestamp‚1.788467266174366e+09 2loss/SRqqt1/assignment_loss‚1.7364857254017676 1loss/FBt3/assignment_loss‚0.05316441245917937 1loss/SRqqt4/detection_loss‚2.3090285807847977 2loss/SRqqt3/assignment_loss‚0.9232308508240079 /loss/FRt1/detection_loss‚2.5994633361697197 /loss/FRt3/detection_loss‚1.7364855638903673 1loss/SRqqt1/detection_loss‚1.7786054382366794 .loss/FBt1/detection_loss‚2.624999666213989 /loss/FBt3/detection_loss‚0.6704988970037769 _step‚229¸ä3’àÙ 0loss/FRt2/assignment_loss‚3.6606765181519263 0loss/FRt3/assignment_loss‚0.9674500148804461 0loss/FRt4/assignment_loss‚3.0347230125356606 0loss/FBt2/assignment_loss‚0.6614197936225017 .loss/FBt4/detection_loss‚2.511816886564096 &loss/total_loss‚44.428071145549204 trainer/global_step‚2799 0loss/FRt1/assignment_loss‚2.6881260921557746 /loss/FRt4/detection_loss¯Îš’Í‚2.8627821648562395 0loss/SRqqt3/detection_loss‚0.961080459093577 epoch‚2 2loss/SRqqt4/assignment_loss‚1.5878309574392107 1loss/FBt4/assignment_loss‚0.17210401103602635 2loss/SRqqt2/assignment_loss‚1.9905798316001893 0loss/SRqqt2/detection_loss‚2.674694974081857 .loss/FRt2/detection_loss‚2.550875496804216 _runtime‚3319.359097563 /loss/FBt1/assignment_loss‚0.513518924242544 .loss/FBt2/detection_loss‚3.158429537500654 $ _timestamp‚1.788467266174366e+09 2loss/SRqqt1/assignment_loss‚1.7364857254017676 1loss/FBt3/assignment_loss‚0.05316441245917937 1loss/SRqqt4/detection_loss‚2.3090285807847977 2loss/SRqqt3/assignment_loss‚0.9232308508240079 /loss/FRt1/detection_loss‚2.5994633361697197 /loss/FRt3/detection_loss‚1.7364855638903673 1loss/SRqqt1/detection_loss‚1.7786054382366794 .loss/FBt1/detection_loss‚2.624999666213989 /loss/FBt3/detection_loss‚0.6704988970037769 _step‚229åˆ23‘á‚(:† Ì°çÔË­ˆ gpu.0.gpu‚39.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚138.302' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.5755 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gý%:}â‚(:ò Ì°çÔ€µÝˆ network.recv‚ 1.089312e+06 disk.dm-0.in‚528395.06640625 disk.md0p1.in‚ 7.12109375* proc.memory.availableMB‚10976.48828125# proc.memory.rssMB‚ 3051.61328125! disk.sdb2.out‚360.69384765625 disk.rbd2.in‚0" disk.nvme1n1.out‚ 28962.0078125 disk.sda.in‚3.625# disk.nvme0n1.in‚328227.71484375* proc.memory.percent‚18.625569343566895 proc.cpu.threads‚17" disk.nvme0n1.out‚ 14203.8515625$ disk./.usageGB‚2974.667121887207 disk.sdb.out‚360.69384765625 disk.dm-0.out‚ 43165.859375$ memory_percent‚33.00483226776123 cpu‚25.124402235874584) disk./.usagePercent‚83.17293772287785 disk.rbd0.in‚0.015625! disk.sda2.out‚360.69384765625 disk.sdb2.in‚ 3.49609375 disk.md0.out‚ 348.1015625 disk.rbd0.out‚ 0.837890625 disk.sdb.in‚ 3.49609375 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda.out‚360.69384765625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚3.625 disk.md0.in‚ 7.12109375 disk.rbd2.out‚ 0.85009765625" disk.nvme1n1.in‚200167.3515625 disk.md0p1.out‚ 348.1015625 network.sent‚ 5.99449e+06ôØ‘ã‚(:† Û°çÔ¤ý͇ gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚134.298' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.5745 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×Ô}òä‚(:ô Û°çÔ°•ïˆ) disk./.usagePercent‚83.17290871121311* proc.memory.percent‚18.625569343566895 disk.sda1.out‚0 disk.sdb1.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.md0p1.in‚ 7.12109375! disk.sdb2.out‚362.48291015625 disk.dm-0.out‚ 43474.9765625$ memory_percent‚33.00483226776123 cpu‚25.12399632897615$ disk./.usageGB‚2974.666084289551# disk.nvme1n1.out‚29271.01171875 disk.sdb.out‚362.48291015625 disk.sda.in‚3.625# disk.nvme0n1.out‚14203.96484375* proc.memory.availableMB‚10976.48828125 disk.rbd2.out‚ 0.85009765625 disk.sda.out‚362.48291015625# disk.nvme0n1.in‚328227.71484375 network.recv‚ 1.094361e+06 disk.sda2.in‚3.625! disk.sda2.out‚362.48291015625 disk.md0p1.out‚ 349.8203125 disk.rbd0.out‚ 0.837890625 disk.rbd2.in‚0" disk.nvme1n1.in‚200167.3515625 disk.dm-0.in‚528395.06640625 disk.sdb.in‚ 3.49609375 disk.sdb1.in‚0 network.sent‚ 6.09083e+06 disk.sdb2.in‚ 3.49609375 disk.md0.in‚ 7.12109375 disk.md0.out‚ 349.8203125 disk.rbd0.in‚0.015625# proc.memory.rssMB‚ 3051.61328125gƜޑå‚(:† ê°çÔ«°”‡ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚104.826' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚26.2065 gpu.0.smClock‚1215 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢ñ;ýyæ‚(:î ê°çÔ¿õ‡ disk.md0.out‚ 351.453125 disk.rbd2.in‚0# disk.nvme1n1.out‚29558.41796875 disk.sda.in‚3.625 disk.nvme0n1.in‚ 328227.90625 disk.md0p1.in‚ 7.12109375# proc.memory.rssMB‚ 3051.61328125! disk.sda2.out‚364.17236328125 disk.sdb.in‚ 3.49609375# disk.nvme0n1.out‚14204.15234375 disk.dm-0.out‚ 43762.5703125* proc.memory.percent‚18.625569343566895 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda.out‚364.17236328125 network.sent‚ 6.097226e+06 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 0.837890625 disk.sda1.out‚0 disk.md0p1.out‚ 351.453125$ memory_percent‚33.00638198852539$ disk./.usageGB‚2975.166084289551 cpu‚25.123536052315018 proc.cpu.threads‚17) disk./.usagePercent‚83.18688892047938! disk.sdb2.out‚364.17236328125 disk.rbd2.out‚ 0.85009765625" disk.nvme1n1.in‚200167.3515625 disk.sda2.in‚3.625 disk.sdb2.in‚ 3.49609375 disk.md0.in‚ 7.12109375 disk.sdb.out‚364.17236328125 disk.dm-0.in‚528395.2578125 disk.sda1.in‚0( proc.memory.availableMB‚ 10976.234375 network.recv‚ 1.098757e+06¯ž@¢’ç‚(:‡ ù°çÔ¶àɇ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚118.751' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.68775 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÂãxÂwè‚(:ì ù°çÔÁäû‡ disk.sda.out‚365.5205078125 disk.sdb2.in‚ 3.49609375( proc.memory.availableMB‚ 10976.484375* proc.memory.percent‚18.625569343566895 network.recv‚ 1.102375e+06 disk.dm-0.out‚ 43839.1875 disk.md0p1.out‚ 352.73828125 disk.sdb2.out‚365.5205078125 disk.rbd0.in‚0.015625# proc.memory.rssMB‚ 3051.61328125) disk./.usagePercent‚83.18689052038735 disk.sdb.out‚365.5205078125 disk.sda.in‚3.625 disk.rbd2.in‚0" disk.nvme1n1.in‚200167.3515625# disk.nvme1n1.out‚29634.83984375 disk.sdb1.out‚0 disk.sda1.out‚0 disk.sda2.out‚365.5205078125 disk.md0.in‚ 7.12109375$ memory_percent‚33.00485610961914 network.sent‚ 6.102122e+06 disk.sda1.in‚0 disk.md0p1.in‚ 7.12109375 disk.md0.out‚ 352.73828125 disk.rbd0.out‚ 0.837890625 cpu‚25.123138264098717 proc.cpu.threads‚17 disk.rbd2.out‚ 0.85009765625% disk./.usageGB‚2975.1661415100098 disk.sdb1.in‚0 disk.nvme0n1.in‚ 328227.90625# disk.nvme0n1.out‚14204.34765625 disk.dm-0.in‚528395.2578125 disk.sdb.in‚ 3.49609375 disk.sda2.in‚3.625*M‹½2é- _wandbruntime‚3375  _runtime‚3375¥@xÇêÁ _runtime‚3375.543998252 _step‚230 & lr-AdamW/pg1‚0.0007790976128583562 & lr-AdamW/pg2‚0.0007790976128583562 trainer/global_step‚2849 % _timestamp‚1.7884673223590527e+09Iu>-ÌëÆ & lr-AdamW/pg1‚0.0007790976128583562 & lr-AdamW/pg2‚0.0007790976128583562 trainer/global_step‚2849 % _timestamp‚1.7884673223590527e+09 _runtime‚3375.543998252 _step‚230æ¶Ô b2ì- _wandbruntime‚3375  _runtime‚3375T Ü'íÖO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131411735 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚704 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52787804 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20521454 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54034373 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54034371 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 % _timestamp‚1.7884673223600395e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5656963260928 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77403707 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5754883516416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5657196929024 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚122641995051520 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5657196929024 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚121428429119488 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚110195623453872 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚121428177448448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131438080 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚115771214187520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20522156 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5754597090816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32266352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚128396902508032 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚115771232190464 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚128396592142336 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚115836799596227 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚110195640493232 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚122642018991616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 trainer/global_step‚2849 _runtime‚3375.545113817‚€ á'îÛO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131438080 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚115771214187520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20522156 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5754597090816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32266352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚128396902508032 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚115771232190464 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚128396592142336 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚115836799596227 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚110195640493232 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚122642018991616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 trainer/global_step‚2849 _runtime‚3375.545113817 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131411735 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚704 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52787804 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20521454 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54034373 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54034371 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 % _timestamp‚1.7884673223600395e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5656963260928 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77403707 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5754883516416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5657196929024 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚122641995051520 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5657196929024 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚121428429119488 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚110195623453872 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚121428177448448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582粿2ï- _wandbruntime‚3376  _runtime‚3376/?¥ðàM UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54052646 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚122683702961664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54052646 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20528010 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚115810518227456 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚115810500224512 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77404056 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5756881731584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20528717 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296898048 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚110233153811296 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5643117592189 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 MCDeviceStatsMoC–œDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54052648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5659160968192 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚121469427511296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5658927286784 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131483073 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52805162 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131456702 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32277152 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216ÙÉ]Ÿë&ñåM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚707 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54052648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5659160968192 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚121469427511296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5658927286784 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131483073 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52805162 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131456702 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32277152 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54052646 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚122683702961664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54052646 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20528010 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚115810518227456 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚115810500224512 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77404056 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5756881731584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20528717 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296898048 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚110233153811296 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5643117592189 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5756584833536 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131483073 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P¡ ÌýÆ & lr-AdamW/pg1‚0.0007763457907628187 & lr-AdamW/pg2‚0.0007763457907628187 trainer/global_step‚2899 % _timestamp‚1.7884673796174517e+09 _runtime‚3432.802505304 _step‚234êæŠ}V2þ- _wandbruntime‚3432  _runtime‚3432‚6À¡}$ÿÔO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5755412635648 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5854795036672 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78741659 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20867521 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚695 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚117811009487495 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚112072158364432 $ _timestamp‚1.788467379618505e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53673971 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54948123 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5755178967552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚124726217494016 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78715316 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5755178967552 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚112072141325072 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚117811255507935 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚123490695004672 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚117735534040064 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚117735516037120 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54948121 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78715316 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32806452 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚123490946675712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚697 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133663437 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54948121 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚123490946675712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133689782 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54948123 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365696 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚123490695004672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78741659 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5755412635648 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/seg©é´Ó]ment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32806450 trainer/global_step‚2899 _step‚235 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5854508611072 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5739097143503 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚130580702164992 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚124726193553920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚ 3432.80364286å¨)ß'€ÙO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78715316 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32806452 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚123490946675712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚697 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133663437 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54948121 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚123490946675712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133689782 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54948123 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365696 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚123490695004672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78741659 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5755412635648 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32806450 trainer/global_step‚2899 _step‚235 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5854508611072 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5739097143503 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚130580702164992 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚124726193553920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚ 3432.80364286 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5755412635648 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5854795036672 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78741659 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20867521 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚695 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚117811009487495 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚112072158364432 $ _timestamp‚1.788467379618505e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53673971 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54948123 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5755178967552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚124726217494016 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78715316 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5755178967552 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚112072141325072 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚117811255507935 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚123490695004672 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚117735534040064 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚117735516037120 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54948121 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608ëþØü¿2- _wandbruntime‚3433  _runtime‚3433^ƒËå&‚ßM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚124767901464064 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54966398 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133734775 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5856545540608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚2899 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚704 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚130624733416960 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚123532196751872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚117774820077056 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚123532196751872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚130624423064576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _runtime‚3433.979370598 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚112109688721856 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚112109671682496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5757376674816 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53691728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5740826651409 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5757142993408 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚117774802074112 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚117850498333905 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ˆöê&ƒäM 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32817252 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20875178 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚124767901464064 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54966398 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133734775 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5856545540608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚2899 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚704 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚130624733416960 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚123532196751872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚117774820077056 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚123532196751872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚130624423064576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _runtime‚3433.979370598 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚112109688721856 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚112109671682496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5757376674816 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53691728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5740826651409 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5757142993408 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚117774802074112 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚117850498333905 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚119699817886720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚119785465399203 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 135941484 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚126810392056320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚113948676235632 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚119699835889664 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80053268 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚125553212560896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚119699835889664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚126810415996416 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5853628342272 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 trainer/global_step‚2949 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5853628342272 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚119699817886720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚119785465399203 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 135941484 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚126810392056320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚113948676235632 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚119699835889664 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80053268 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚125553212560896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚119699835889664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0ï-E©h2•- _wandbruntime‚3491  _runtime‚3491ù æ"–àM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80079960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135986477 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54579613 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚694 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚125594714308096 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119824954245613 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚125594462623744 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33357352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚126852076026368 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80106329 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80079960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚119739121926656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135960106 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5838993673205 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21221565 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚132808541869568 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚126852099966464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217984 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5838764691917 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 I?Dú¯‰ÄeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5855358700032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 135986477 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55880148 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5855358700032 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21222259 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80106329 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55880148 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚696 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5956465843200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5855592381440 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54578917 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55880146 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚113986206593056 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _runtime‚3491.750813507 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2949 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55880146 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚119825200266261 _step‚240ÙaAë&—åM F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5855592381440 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54578917 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55880146 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚113986206593056 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _runtime‚3491.750813507 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2949 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55880146 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚119825200266261 _step‚240 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80079960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135986477 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54579613 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚694 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚125594714308096 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119824954245613 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚125594462623744 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33357352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚126852076026368 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80106329 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80079960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚119739121926656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135960106 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5838993673205 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21221565 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚132808541869568 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚126852099966464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217984 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5838764691917 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5855358700032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 135986477 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55880148 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5855358700032 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21222259 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80106329 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55880148 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚696 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5956465843200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0ð¯±å2˜- _wandbruntime‚3491  _runtime‚3491D‰}ó\™Ö /loss/FRt4/assignment_loss‚3.827782306406233 1loss/FBt1/assignment_loss‚0.05241804967821233 1loss/SRqqt4/detection_loss‚2.4832398540207317 .loss/FBt4/detection_loss‚1.700562192996343 0loss/FRt3/assignment_loss‚1.1801254581660032 1loss/SRqqt1/assignment_loss‚1.583385425971614 0loss/SRqqt3/detection_loss‚0.763443736568763 /loss/FBt1/detection_loss‚1.9266470074653625 /loss/FBt3/detection_loss‚0.5931027920657321 _step‚241 3loss/FBt3/assignment_loss‚0.0038662970450945113 _runtime‚3491.751725171 0loss/FRt2/assignment_loss‚3.6467828691005706 1loss/SRqqt4/assignment_loss‚1.965369489043951 /loss/FRt3/detection_loss‚1.5943982714088634 1loss/SRqqt1/detection_loss‚1.9839750740263198 0loss/SRqqt2/detection_loss‚2.476341212237323 $ _timestamp‚1.788467438566988e+09 .loss/FBt2/detection_loss‚3.445468020439148 %loss/total_loss‚44.71248957579249 2loss/SRqqt2/assignment_loss‚1.9954589185891327 1loss/FBt2/assignment_loss‚0.01630051237734733 1loss/FBt4/assignment_loss‚0.15359859623834685 /loss/FRt1/detection_loss‚2.9470972660928965 /loss/FRt4/detection_loss‚3.5679106414318085 epoch‚2 trainer/global_step‚2949 0loss/FRt1/assignment_loss‚3.3264670483767986 .loss/FRt2/detection_loss‚2.272962873817087 2loss/SRqqt3/assignment_loss‚1.2057856622288066ok§çašÛ /loss/FBt1/detection_loss‚1.9266470074653625 /loss/FBt3/detection_loss‚0.5931027920657321 _step‚241 3loss/FBt3/assignment_loss‚0.0038662970450945113 _runtime‚3491.751725171 0loss/FRt2/assignment_loss‚3.6467828691005706 1loss/SRqqt4/assignment_loss‚1.965369489043951 /loss/FRt3/detection_loss‚1.5943982714088634 1loss/SRqqt1/detection_loss‚1.9839750740263198 0loss/SRqqt2/detection_loss‚2.476341212237323 $ _timestamp‚1.788467438566988e+09 .loss/FBt2/detection_loss‚3.445468020439148 %loss/total_loss‚44.71248957579249 2loss/SRqqt2/assignment_loss‚1.9954589185891327 1loss/FBt2/assignment_loss‚0.01630051237734733 1loss/FBt4/assignment_loss‚0.15359859623834685 /loss/FRt1/detection_loss‚2.9470972660928965 /loss/FRt4/detection_loss‚3.5679106414318085 epoch‚2 trainer/global_step‚2949 0loss/FRt1/assignment_loss‚3.3264670483767986 .loss/FRt2/detection_loss‚2.272962873817087 2loss/SRqqt3/assignment_loss‚1.2057856622288066 /loss/FRt4/assignment_loss‚3.827782306406233 1loss/FBt1/assignment_loss‚0.05241804967821233 1loss/SRqqt4/detection_loss‚2.4832398540207317 .loss/FBt4/detection_loss‚1.700562192996343 0loss/FRt3/assignment_loss‚1.1801254581660032 1loss/SRqqt1/assignment_loss‚1.583385425971614 0loss/SRqqt3/detection_loss‚0.763443736568763ñ606’›‚(:‡ ñ±çÔÐÖЇ gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚144.821' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.20525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Äœ9Jsœ‚(:è ñ±çÔí»ˆ) disk./.usagePercent‚83.16997042687422 disk.sdb2.in‚ 3.49609375 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 0.837890625% memory_percent‚33.007264137268066 disk.sda2.in‚3.625 network.sent‚ 6.362335e+06% disk./.usageGB‚2974.5609970092773 disk.dm-0.out‚45813.95703125 disk.md0.out‚ 365.40234375 disk.rbd2.in‚0 disk.sdb.out‚378.6591796875 disk.sda.in‚3.625 disk.sdb2.out‚378.6591796875 disk.sda2.out‚378.6591796875 disk.md0p1.in‚ 7.12109375# disk.nvme0n1.out‚14205.18359375 disk.md0p1.out‚ 365.40234375# proc.memory.rssMB‚ 3051.61328125 cpu‚25.1199347881349 disk.md0.in‚ 7.12109375" disk.nvme1n1.in‚200184.4609375 disk.sdb.in‚ 3.49609375 disk.sdb1.in‚0 disk.nvme0n1.in‚328243 disk.sda1.in‚0* proc.memory.availableMB‚10976.08984375* proc.memory.percent‚18.625569343566895 disk.rbd2.out‚ 0.99951171875 disk.sdb1.out‚0 disk.sda.out‚378.6591796875 proc.cpu.threads‚17 network.recv‚ 1.1374e+06" disk.nvme1n1.out‚ 31608.7734375 disk.dm-0.in‚528427.4609375 disk.sda1.out‚0ýî̈’‚(:‡ €²çÔªÔŽ† gpu.0.gpu‚14.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚142.313' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.57825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Э³Ïož‚(:ä €²çÔš¼€‡! disk.nvme1n1.out‚ 31779.390625 disk.nvme0n1.in‚328243 disk.sda1.out‚0 disk.sda2.in‚3.625 disk.rbd2.out‚1.0625 disk.sda.in‚3.625 disk.rbd0.out‚ 0.837890625$ memory_percent‚33.00731182098389 network.recv‚ 1.142881e+06) disk./.usagePercent‚83.15034158248733 disk.sdb.in‚ 3.49609375 disk.sdb1.out‚0 disk.rbd0.in‚0.015625" disk.nvme1n1.in‚200184.4609375 disk.sdb1.in‚0 disk.sda.out‚380.24560546875" disk.nvme0n1.out‚ 14205.2421875* proc.memory.availableMB‚10976.08203125* proc.memory.percent‚18.625569343566895 disk.dm-0.out‚ 45984.6328125 disk.sda1.in‚0 disk.md0.out‚ 366.94140625 cpu‚25.119519420091105 network.sent‚ 6.459409e+06$ disk./.usageGB‚2973.858974456787 disk.dm-0.in‚528427.4609375! disk.sda2.out‚380.24560546875 disk.md0p1.out‚ 366.94140625 disk.sdb2.in‚ 3.49609375# proc.memory.rssMB‚ 3051.61328125 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sdb.out‚380.24560546875 disk.md0p1.in‚ 7.12109375! disk.sdb2.out‚380.24560546875 disk.md0.in‚ 7.12109375—®Š;‘Ÿ‚(:† ²çÔ½¢Á‡ gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚145.098' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.2745 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I(þp ‚(:å ²çÔû¹Ù‡ disk.sda.in‚3.625 disk.dm-0.out‚ 46131.6796875! disk.sda2.out‚382.17724609375 disk.sdb1.in‚0# proc.memory.rssMB‚ 3051.61328125 network.recv‚ 1.146728e+06 disk.sda1.in‚0 disk.md0p1.out‚ 368.80859375* proc.memory.percent‚18.625569343566895 disk.sda.out‚382.17724609375 disk.dm-0.in‚528427.4609375" disk.nvme1n1.in‚200184.4609375 cpu‚25.119145845378824 proc.cpu.threads‚17$ disk./.usageGB‚2974.109519958496 disk.nvme0n1.in‚328243 disk.sdb2.in‚ 3.49609375 disk.md0.out‚ 368.80859375 disk.rbd2.in‚0# disk.nvme1n1.out‚31926.23828125 disk.sdb.in‚ 3.49609375$ memory_percent‚33.00504684448242# disk.nvme0n1.out‚14205.44140625 disk.sdb.out‚382.17724609375 disk.sdb1.out‚0( proc.memory.availableMB‚ 10976.453125 network.sent‚ 6.464808e+06 disk.sda1.out‚0 disk.md0.in‚ 7.12109375 disk.rbd0.out‚ 0.837890625 disk.rbd2.out‚1.0625) disk./.usagePercent‚83.15734693957654 disk.md0p1.in‚ 7.12109375 disk.sda2.in‚3.625! disk.sdb2.out‚382.17724609375 disk.rbd0.in‚0.015625Îÿ@‘¡‚(:† ž²çÔ­µÃ‡ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚101.986' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.4965 gpu.0.smClock‚1170 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º3æj¢‚(:ß ž²çÔùÎ¥ˆ disk.sda.in‚3.625 disk.sda1.in‚0! disk.sda2.out‚383.72509765625 disk.sdb2.in‚ 3.49609375* proc.memory.availableMB‚10976.44921875 proc.cpu.threads‚17 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 0.837890625 disk.dm-0.out‚46219.76953125 disk.sda2.in‚3.625$ memory_percent‚33.00507068634033 network.recv‚ 1.151329e+06 disk.rbd2.out‚1.0625 disk.md0p1.in‚ 7.12109375 disk.md0.in‚ 7.12109375) disk./.usagePercent‚83.15729478257634$ disk./.usageGB‚2974.107654571533" disk.nvme1n1.in‚200184.4609375# disk.nvme1n1.out‚32014.26953125 disk.sdb1.in‚0 disk.md0p1.out‚ 370.3046875 disk.md0.out‚ 370.3046875 disk.sdb1.out‚0 disk.sda.out‚383.72509765625 disk.nvme0n1.out‚14205.5 disk.dm-0.in‚528427.4609375! disk.sdb2.out‚383.72509765625# proc.memory.rssMB‚ 3051.61328125 disk.rbd2.in‚0 cpu‚25.118706911987136 network.sent‚ 6.471151e+06 disk.sdb.out‚383.72509765625 disk.nvme0n1.in‚328243 disk.sda1.out‚0* proc.memory.percent‚18.625569343566895 disk.sdb.in‚ 3.49609375š+²2£- _wandbruntime‚3542  _runtime‚3542â$é¸Ç¤Á & lr-AdamW/pg1‚0.0007706150900245984 & lr-AdamW/pg2‚0.0007706150900245984 trainer/global_step‚2999 % _timestamp‚1.7884674895596602e+09 _runtime‚3542.744570661 _step‚242áVøXÌ¥Æ _step‚242 & lr-AdamW/pg1‚0.0007706150900245984 & lr-AdamW/pg2‚0.0007706150900245984 trainer/global_step‚2999 % _timestamp‚1.7884674895596602e+09 _runtime‚3542.744570661ò²´m2¦- _wandbruntime‚3542  _runtime‚3542üèý1ë§ÖO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56775623 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 trainer/global_step‚2999 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚121664119736320 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5951844048896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 % _timestamp‚1.7884674895606337e+09 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56775623 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5934973224519 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚128894614498816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚121759921310911 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56775621 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚3542.745665377 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5934744243439 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚121760167331351 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56775621 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚121664137739264 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5951844048896 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚121664137739264 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚128894590558720 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6054684345856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81417563 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81391220 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5951610380800 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚127615730117120 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55446486 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚127615730117120 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21560538 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚121664119736320 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚127615981788160 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33886652 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21559836 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56775623 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 trainer/global_step‚2999 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚121664119736320 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5951844048896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 % _timestamp‚1.7884674895606337e+09 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56775623 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5934973224519 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚128894614498816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚121759921310911 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56775621 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚3542.745665377 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5934744243439 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚121760167331351 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56775621 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚121664137739264 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5951844048896 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚121664137739264 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚128894590558720 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6054684345856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81417563 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81391220 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5951610380800 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚127615730117120 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55446486 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚127615730117120 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21560538 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚121664119736320 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚127615981788160 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33886652ó3YÄ2©- _wandbruntime‚3543  _runtime‚3543˜\så&ªßM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚121799410157321 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚679 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134992654950912 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 303189504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚3543.687025091 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138238179 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚244 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚127656980179968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81444281 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5936931713713 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚2999 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 138211808 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788467490501378e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5953574406656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚121799656177969 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138211808 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81417912 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚121703405773312 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81417912 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56793898 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21566517 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55463969 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56793896 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33897452 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55464650 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 327129600 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128936274528768 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5953808088064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273mQê&«äM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81444281 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5936931713713 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚2999 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 138211808 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788467490501378e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5953574406656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚121799656177969 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138211808 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81417912 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚121703405773312 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81417912 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56793898 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21566517 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55463969 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56793896 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33897452 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55464650 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 327129600 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128936274528768 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5953808088064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚121799410157321 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚679 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134992654950912 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 303189504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚3543.687025091 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138238179 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚244 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚127656980179968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚129678499344384 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚117701694938672 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚130978789061120 _step‚247 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82729172 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚694 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6154655018496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57689373 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚129678499344384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚130978813001216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚129678247673344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚123628421585920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57689373 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚123628439588864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 trainer/global_step‚3049 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚137133159751168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚137133468019712 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129678247673344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005euˆvá'ºÛO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82755515 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚130978789061120 _step‚247 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82729172 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚694 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6154655018496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57689373 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚129678499344384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚130978813001216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚129678247673344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚123628421585920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57689373 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚123628439588864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 trainer/global_step‚3049 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚137133159751168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚137133468019712 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129678247673344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 140418543 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6050059755520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6050059755520 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚692 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚123628439588864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56333635 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6154370690048 % _timestamp‚1.7884675417421842e+09 _runtime‚3594.927227621 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚129678499344384 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚117701694938672 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚123773866069029 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6051790113280 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34437552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚665 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6156651231232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚117739225296096 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57707648 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82782233 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57707648 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57707646 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚123667707622912 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6052023794688 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 _runtime‚3595.87ÎKl½87390415 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚129719497736192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21914609 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82755864 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _step‚248 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140489881 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚117739242335456 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56351496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140489881 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 140463510 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82782233 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚123667725625856 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚129719749420544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6156375304704 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6052023794688 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21913944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275926528 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚123774112089677 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚137177148202496 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0†ø°ië&½åM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57707648 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82782233 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57707648 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57707646 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚123667707622912 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6052023794688 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 _runtime‚3595.877390415 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚129719497736192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21914609 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82755864 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _step‚248 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140489881 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚117739242335456 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56351496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140489881 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 140463510 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82782233 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚123667725625856 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚129719749420544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6156375304704 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6052023794688 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21913944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275926528 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚123774112089677 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚137177148202496 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 % _timestamp‚1.7884675426915572e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚129719749420544 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚123773866069029 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6051790113280 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34437552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚665 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6156651231232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚117739225296096 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131740765229568 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚125592723435520 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚133063011503616 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58603123 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚251 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34966850 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57219589 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚119578212809872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84067124 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚125709079154767 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚125592723435520 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚119578229849232 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚125592741438464 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚125708833134327 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6130849305535 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84093467 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚133062987563520 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22253401 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚139317377633280 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328448 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6254390069760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582Ÿ³9­à'ÎÚO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚3099 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚131740765229568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58603123 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚251 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34966850 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57219589 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚119578212809872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84067124 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚125709079154767 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚125592723435520 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚119578229849232 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚125592741438464 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚125708833134327 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6130849305535 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84093467 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚133062987563520 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22253401 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚139317377633280 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328448 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6254390069760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84067124 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6148275462144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57220253 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6254674398208 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 $ _timestamp‚1.788467592882184e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131740765229568 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚125592723435520 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚133063011503616 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6132807794729 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 142741583 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142741583 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚663 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34977654 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22259971 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6256671685632 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚125748568001385 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 142715212 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚125632027475456 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6132578813441 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57237623 trainer/global_step‚3099 % _timestamp‚1.7884675938286576e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84093816 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚125632027475456 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6150239501312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34977652 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚131782266976768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6150005819904 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57238286 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚125748321980737 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58621396 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 304060928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚125632009472512 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚119615760206656 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280120832 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84120185 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚119615743167296 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58621398 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚3647.014885223 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚131782266976768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚133104671533568 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚125632009472512 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚139361367159296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6256391564800 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58621398 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚133104695473664 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142741583 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚663 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34977654 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22259971 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6256671685632 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚125748568001385 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 142715212 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚125632027475456 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6132578813441 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57237623 trainer/global_step‚3099 % _timestamp‚1.7884675938286576e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84093816 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚125632027475456 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6150239501312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34977652 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚131782266976768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6150005819904 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57238286 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚125748321980737 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58621396 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 304060928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚125632009472512 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚119615760206656 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280120832 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84120185 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚119615743167296 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58621398 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚3647.014885223 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚131782266976768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚133104671533568 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚125632009472512 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚139361367159296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6256391564800 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58621398 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚133104695473664 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6132807794729 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 142741583 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚133803534456832 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6246257500672 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58106404 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35506952 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚127557025285120 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚141501564136448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35506950 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚127557025285120 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚688 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 144948292 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 trainer/global_step‚3149 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚135147186065920 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚127683289046035 _runtime‚3699.529338882 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚141501882890752 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294814208 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85431419 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6354378070528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6246257500672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85405076 % _timestamp‚1.7884676463442516e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59516873 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85405076 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚133803534456832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚121454730681072 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚133803282785792 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58107094 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59516871 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22599454 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6228787346043 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318754304 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 _step‚255 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144921947 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6354672884736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6228558364963 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133803282785792 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133803282785792 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚133803534456832 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6246257500672 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58106404 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35506952 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚127557025285120 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚141501564136448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35506950 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚127557025285120 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚688 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 144948292 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 trainer/global_step‚3149 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚135147186065920 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚127683289046035 _runtime‚3699.529338882 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚141501882890752 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294814208 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85431419 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6354378070528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6246257500672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85405076 % _timestamp‚1.7884676463442516e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59516873 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85405076 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚133803534456832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚121454730681072 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚133803282785792 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58107094 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59516871 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22599454 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6228787346043 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0ÿ;Ã32á- _wandbruntime‚3700  _runtime‚3700‡çW–âßM ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚121492278077856 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚127596329325056 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 $ _timestamp‚1.788467647311348e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59535148 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3149 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6230745835237 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚141545267275776 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚133844784532992 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6248455207936 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚133844784532992 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85431768 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6248221526528 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59535146 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35517754 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚127723023913093 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚135188870035968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6356397239808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitÞÃbOor.on_train_batch_end/segment.all.allocated‚582 _runtime‚3700.497116351 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚127596311322112 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58124631 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306158080 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚135188893976064 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6248455207936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 144966914 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚121492261038496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 _step‚256 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22606188 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217984 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚127596311322112 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35517752 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6356679457792 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 144993285 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58123940 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚133844532848640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59535146 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85458137 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59535148 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22606877 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚689 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚127722777892445 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6230516853949 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6248221526528 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚141545573433856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85431768 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚127596329325056 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144993285 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0®^ ×ê&ãäM L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144993285 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚121492278077856 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚127596329325056 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 $ _timestamp‚1.788467647311348e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59535148 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3149 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6230745835237 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚141545267275776 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚133844784532992 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6248455207936 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚133844784532992 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85431768 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6248221526528 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59535146 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35517754 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚127723023913093 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚135188870035968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6356397239808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 _runtime‚3700.497116351 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚127596311322112 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58124631 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306158080 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚135188893976064 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6248455207936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 144966914 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚121492261038496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 _step‚256 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22606188 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217984 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚127596311322112 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35517752 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6356679457792 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 144993285 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58123940 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚133844532848640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59535146 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85458137 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59535148 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22606877 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚689 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚127722777892445 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6230516853949 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6248221526528 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚141545573433856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85431768 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚127596329325056 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36047052 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60430623 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86769371 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 147173649 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 _runtime‚3753.816049713 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6344706875392 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚129657990978183 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚143686018931200 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚123331265591632 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚135865800342016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _step‚259 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚123331248552272 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6326725386ÀcîD} 551 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚699 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58993579 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 % _timestamp‚1.7884677006308074e+09 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚143685706468352 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚135865800342016 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129521327134720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22947226 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚137231384568320 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 147199994 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚129657744957743íƒU¾á'ôÛO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚123331248552272 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6326725386551 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚699 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58993579 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 % _timestamp‚1.7884677006308074e+09 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚143685706468352 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚135865800342016 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129521327134720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22947226 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚137231384568320 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 147199994 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚129657744957743 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚135866052013056 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86743028 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚137231408508416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚135866052013056 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6454610422784 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129521345137664 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6344473207296 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129521345137664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚697 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6454321900032 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60430623 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚3199 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86743028 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36047052 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60430623 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86769371 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 147173649 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 _runtime‚3753.816049713 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6344706875392 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚129657990978183 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚143686018931200 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚123331265591632 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚135865800342016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _step‚259ƒéõ«v2õ- _wandbruntime‚3754  _runtime‚3754óª/¥å&ößM QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86796089 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86769720 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚135907302089216 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6346437233152 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147244987 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6328683875745 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚135907050404864 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚135907302089216 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 147244987 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚129697479824801 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚123368795949056 $ _timestamp‚1.788467701680788e+09 _runtime‚3754.866767216 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86769720 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59011689 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚260 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60448896 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59012364 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚137273068538368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 147218616 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 147218616 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86796089 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚123368778909696 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60448898 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22953837 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6456606329856 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296898048 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6456309431808 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚129697233804153 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6346670914560 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚143729698808320 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 trainer/global_step‚3199 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚129560631174656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60448896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚137273092478464 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60448898 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320838144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6346437233152 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147244987 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6328683875745 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚135907050404864 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚135907302089216 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 147244987 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚129697479824801 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚123368795949056 $ _timestamp‚1.788467701680788e+09 _runtime‚3754.866767216 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86769720 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59011689 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚260 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60448896 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59012364 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚137273068538368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 147218616 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 147218616 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86796089 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚123368778909696 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60448898 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22953837 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6456606329856 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296898048 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6456309431808 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚129697233804153 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6346670914560 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚143729698808320 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 trainer/global_step‚3199 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚129560631174656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60448896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚137273092478464 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60448898 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320838144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61344371 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚3808.585053229 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚139315583070720 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61344371 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328448 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 149425351 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚131485646987264 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23294076 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚131485628984320 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚137928317898240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61344373 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚145870119074816 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6424434445979 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚686 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚137928569569280 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6442688913920 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88080980 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23294760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚263 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36587150 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚139315607010816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚131485628984320 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚137928317898240 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88107323 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚125207783462832 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚137928569569280 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59881912 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59881226 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36587152 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚131632200869451 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6442922582016}¹á'†ÛO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚145869810806272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚137928317898240 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88107323 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚125207783462832 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚137928569569280 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59881912 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59881226 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36587152 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚131632200869451 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6442922582016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚131632446889891 trainer/global_step‚3249 % _timestamp‚1.7884677554000435e+09 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61344371 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚3808.585053229 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚139315583070720 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61344371 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328448 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 149425351 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚131485646987264 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23294076 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚131485628984320 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚137928317898240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61344373 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚145870119074816 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6424434445979 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚686 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚137928569569280 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6442688913920 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88080980 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23294760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚263 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36587150 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚139315607010816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚131485628984320 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149496689 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61362646 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 _runtime‚3809.519202194 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59898743 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36597954 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚131671935736509 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6444652939776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚145913799712768 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚145913489360384 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6556508731904 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚131671689715861 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149496689 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 \JDeviceStatsMonitor.on_train_batch_end/allocated_b§ú:"Îytes.small_pool.allocated‚ 6444886621184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _step‚264 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6426621916253 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61362646 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6444652939776 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚131524915021312 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884677563337765e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61362648 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚131524933024256 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚3249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚131524915021312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚125245313820256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412288 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36597952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚125245296780896 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88134041 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6556222319616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚131524933024256 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23301474 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582€ì»ë&‰åM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149496689 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61362646 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 _runtime‚3809.519202194 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59898743 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36597954 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚131671935736509 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6444652939776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚145913799712768 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚145913489360384 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6556508731904 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚131671689715861 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149496689 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6444886621184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _step‚264 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6426621916253 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61362646 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6444652939776 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚131524915021312 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884677563337765e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61362648 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚131524933024256 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚3249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚131524915021312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚125245313820256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412288 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36597952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚125245296780896 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88134041 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6556222319616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚131524933024256 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23301474 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149470318 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6444886621184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61362648 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚685 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149470318 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352384 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚139357290980864ˆäœï2Š- _wandbruntime‚3809  _runtime‚3809OKKˆY‹Ó 2loss/SRqqt4/assignment_loss‚1.4090433859306832 /loss/FBt4/detection_loss‚1.8549714638636663 -loss/FBt2/detection_loss‚2.35132081593786 /loss/FBt3/detection_loss‚0.7214932836646187 0loss/FRt1/assignment_loss‚2.7649159845378666 1loss/SRqqt3/detection_loss‚0.9444616304104587 /loss/SRqqt4/detection_loss‚2.87264382061751 epoch‚2 2loss/SRqqt1/assignment_loss‚1.2513112170765033 1loss/SRqqt3/assignment_loss‚1.063786580266588 0loss/FBt4/assignment_loss‚0.9104795799626467 .loss/FRt4/detection_loss‚2.926757407599482 %loss/total_loss‚44.32700718073026 % _timestamp‚1.7884677563360896e+09 2loss/FBt3/assignment_loss‚0.023540045904005255 1loss/SRqqt1/detection_loss‚1.7720136378820126 2loss/SRqqt2/assignment_loss‚1.4955485504444526 0loss/FBt1/assignment_loss‚0.4358495220076293 /loss/FRt3/detection_loss‚1.4558093911858605 _runtime‚3809.520789511 1loss/FBt2/assignment_loss‚0.43912915739097763 0loss/SRqqt2/detection_loss‚2.702605488507644 .loss/FBt1/detection_loss‚3.292340433597565 0loss/FRt4/assignment_loss‚3.8729529791864854 /loss/FRt1/detection_loss‚2.5861658967203565 /loss/FRt2/detection_loss‚2.1178644126927364 trainer/global_step‚3249 _step‚265 /loss/FRt2/assignment_loss‚3.557221743525291 0loss/FRt3/assignment_loss‚1.5047807518173666Ê_\¹^ŒØ /loss/FRt2/assignment_loss‚3.557221743525291 0loss/FRt3/assignment_loss‚1.5047807518173666 2loss/SRqqt4/assignment_loss‚1.4090433859306832 /loss/FBt4/detection_loss‚1.8549714638636663 -loss/FBt2/detection_loss‚2.35132081593786 /loss/FBt3/detection_loss‚0.7214932836646187 0loss/FRt1/assignment_loss‚2.7649159845378666 1loss/SRqqt3/detection_loss‚0.9444616304104587 /loss/SRqqt4/detection_loss‚2.87264382061751 epoch‚2 2loss/SRqqt1/assignment_loss‚1.2513112170765033 1loss/SRqqt3/assignment_loss‚1.063786580266588 0loss/FBt4/assignment_loss‚0.9104795799626467 .loss/FRt4/detection_loss‚2.926757407599482 %loss/total_loss‚44.32700718073026 % _timestamp‚1.7884677563360896e+09 2loss/FBt3/assignment_loss‚0.023540045904005255 1loss/SRqqt1/detection_loss‚1.7720136378820126 2loss/SRqqt2/assignment_loss‚1.4955485504444526 0loss/FBt1/assignment_loss‚0.4358495220076293 /loss/FRt3/detection_loss‚1.4558093911858605 _runtime‚3809.520789511 1loss/FBt2/assignment_loss‚0.43912915739097763 0loss/SRqqt2/detection_loss‚2.702605488507644 .loss/FBt1/detection_loss‚3.292340433597565 0loss/FRt4/assignment_loss‚3.8729529791864854 /loss/FRt1/detection_loss‚2.5861658967203565 /loss/FRt2/detection_loss‚2.1178644126927364 trainer/global_step‚3249 _step‚265‰È.<’‚(:‡ ¬´çÔ§ººˆ gpu.0.gpu‚43.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚132.399' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.09975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01ImŽ‚(:â ¬´çÔÞ©Ùˆ) disk./.usagePercent‚83.13923235476419 disk.md0.in‚ 7.12109375# disk.nvme1n1.out‚35053.74609375 disk.sda.out‚ 413.201171875 network.recv‚ 1.299507e+06 disk.md0.out‚ 398.71875 disk.rbd2.out‚1.0625 disk.sdb.out‚ 413.201171875 disk.md0p1.out‚ 398.71875# proc.memory.rssMB‚ 3051.61328125 cpu‚25.108996402506058 proc.cpu.threads‚17$ disk./.usageGB‚2973.461654663086 disk.rbd0.in‚0.015625 disk.nvme0n1.out‚ 14779.65625 disk.md0p1.in‚ 7.12109375 disk.sdb2.in‚ 3.49609375 disk.sdb2.out‚ 413.201171875 disk.rbd0.out‚ 0.837890625 disk.sdb1.in‚0# disk.nvme0n1.in‚328243.05078125 disk.rbd2.in‚0 disk.sdb1.out‚0* proc.memory.availableMB‚10975.60546875 network.sent‚ 7.112878e+06" disk.nvme1n1.in‚200184.4921875 disk.sdb.in‚ 3.49609375 disk.dm-0.out‚49833.40234375 disk.sda2.out‚ 413.201171875% memory_percent‚33.010220527648926 disk.sda.in‚3.625 disk.sda1.in‚0 disk.sda2.in‚3.625 disk.dm-0.in‚528427.54296875 disk.sda1.out‚0* proc.memory.percent‚18.625569343566895Ù¾.%‘‚(:† »´çÔ«Üö† gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚134.298' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.5745 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬83t‚(:é »´çÔû—ä‡ disk.rbd0.out‚ 0.837890625 disk.sdb1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 3051.61328125 disk.md0.out‚ 399.9609375 disk.rbd2.out‚1.0625# disk.nvme0n1.in‚328243.05078125* proc.memory.availableMB‚10976.08203125 disk.md0.in‚ 7.12109375 disk.rbd2.in‚0 disk.dm-0.out‚49946.09765625 proc.cpu.threads‚17 network.sent‚ 7.221702e+06 disk.sdb2.in‚ 3.49609375* proc.memory.percent‚18.625569343566895 disk.nvme1n1.out‚ 35165.78125 disk.sda.in‚3.625 disk.sda.out‚414.4912109375 disk.sda2.out‚414.4912109375 disk.md0p1.out‚ 399.9609375 disk.sdb2.out‚414.4912109375 disk.sdb.in‚ 3.49609375 disk.sdb1.out‚0 disk.sda2.in‚3.625 disk.md0p1.in‚ 7.12109375 cpu‚25.108714445345004 network.recv‚ 1.305251e+06" disk.nvme1n1.in‚200184.4921875 disk.sda1.in‚0 disk.rbd0.in‚0.015625 disk.sdb.out‚414.4912109375# disk.nvme0n1.out‚14780.31640625 disk.dm-0.in‚528427.54296875$ memory_percent‚33.00731182098389) disk./.usagePercent‚83.14296387348047$ disk./.usageGB‚2973.595111846924j¡’‘‚(:‡ Ê´çÔäÜ»ˆ gpu.0.gpu‚28.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚145.771' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.44275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cë0ºq’‚(:æ Ê´çÔº¶éŠ disk.sda1.in‚0 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.sda.in‚3.625 disk.md0p1.out‚ 401.74609375! disk.sdb2.out‚416.33935546875 cpu‚25.108252105399 disk.sda.out‚416.33935546875 disk.dm-0.in‚528427.54296875! disk.sda2.out‚416.33935546875 disk.sdb2.in‚ 3.49609375 network.recv‚ 1.31007e+06 disk.rbd0.out‚ 0.837890625 disk.sdb1.in‚0$ disk./.usageGB‚2973.596878051758 disk.sdb.out‚416.33935546875 disk.sda2.in‚3.625$ memory_percent‚33.00955295562744# proc.memory.rssMB‚ 3051.61328125* proc.memory.percent‚18.625569343566895 disk.rbd2.in‚0 disk.rbd2.out‚1.0625" disk.nvme1n1.in‚200184.4921875! disk.nvme0n1.out‚ 14851.046875 disk.md0.in‚ 7.12109375 disk.md0.out‚ 401.74609375* proc.memory.availableMB‚10975.71484375 network.sent‚ 7.228096e+06) disk./.usagePercent‚83.14301325730685 disk.sdb.in‚ 3.49609375 disk.sda1.out‚0 disk.md0p1.in‚ 7.12109375 disk.rbd0.in‚0.015625! disk.nvme1n1.out‚ 35165.890625# disk.nvme0n1.in‚328243.05078125 disk.dm-0.out‚ 50016.9375> sœ“‚(:‘ Ù´çÔ¬êׇ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚107.611' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚26.902750000000005 gpu.0.smClock‚1215 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œàçz”‚(:ï Ù´çÔ¬è÷‡ cpu‚25.107873661642714 network.recv‚ 1.31461e+06) disk./.usagePercent‚83.14328385507683 disk.md0p1.out‚ 403.20703125# disk.nvme0n1.in‚328243.05078125 disk.sda1.out‚0! disk.sda2.out‚417.87060546875 disk.sdb.in‚ 3.49609375" disk.nvme0n1.out‚ 14927.6015625# proc.memory.rssMB‚ 3051.61328125 disk.sda1.in‚0 disk.md0.in‚ 7.12109375 disk.sda.out‚417.87060546875 disk.dm-0.in‚528427.54296875 disk.md0p1.in‚ 7.12109375! disk.sdb2.out‚417.87060546875 disk.sdb.out‚417.87060546875* proc.memory.availableMB‚10975.36328125 proc.cpu.threads‚17 disk.rbd2.out‚1.0625% memory_percent‚33.011698722839355 network.sent‚ 7.231697e+06 disk.md0.out‚ 403.20703125 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.sda.in‚3.625% disk./.usageGB‚2973.6065559387207 disk.dm-0.out‚ 50095.0546875 disk.sda2.in‚3.625 disk.rbd0.out‚ 0.837890625* proc.memory.percent‚18.625569343566895 disk.sdb2.in‚ 3.49609375" disk.nvme1n1.in‚200184.4921875! disk.nvme1n1.out‚ 35167.453125 disk.sdb1.in‚0l©í2•- _wandbruntime‚3860  _runtime‚3860¶ðPÆ–À & lr-AdamW/pg1‚0.0007516509251544893 & lr-AdamW/pg2‚0.0007516509251544893 trainer/global_step‚3299 $ _timestamp‚1.788467807341619e+09 _runtime‚3860.526532003 _step‚266L7aAË—Å & lr-AdamW/pg1‚0.0007516509251544893 & lr-AdamW/pg2‚0.0007516509251544893 trainer/global_step‚3299 $ _timestamp‚1.788467807341619e+09 _runtime‚3860.526532003 _step‚266Šîãå“2˜- _wandbruntime‚3860  _runtime‚3860!8u‡Î™ÖO N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89418932 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚148053844195840 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89445275 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6541138288640 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚133449930833920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚133449948836864 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚141399805513216 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6522601467567 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚3299 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚133449930833920 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6654353242624 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23638064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139990835454464 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6540904620544 _step‚267 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151677053 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6522372486487 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚133449948836864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 151703398 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62258123 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62258121 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62258123 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚141399781573120 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚139990835454464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚133606902801599 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6541138288640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6540904620544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚139991087125504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peakü‰qÉ ‚ 315205120 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89445275 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60764604 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89418932 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139990835454464 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6540904620544 _step‚267 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151677053 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6522372486487 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚133449948836864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 151703398 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62258123 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62258121 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62258123 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚141399781573120 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚139990835454464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚133606902801599 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6541138288640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6540904620544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚139991087125504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89445275 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60764604 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89418932 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚148053844195840 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89445275 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6541138288640 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚133449930833920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚133449948836864 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚141399805513216 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6522601467567 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚3299 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚133449930833920 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6654353242624 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23638064‹JŸË2›- _wandbruntime‚3861  _runtime‚3861DÜÛå&œßM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62276396 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23644638 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37138054 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚141441465543168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6656342137344 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 151748391 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚148097831620608 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6524559956761 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 $ _timestamp‚1.788467808277336e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚140032337201664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6524330975473 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60782010 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚141441489483264 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89445624 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6543102327808 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89445624 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62276398 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151748391 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288509440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚3861.463862418 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚133489216870912 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6543102327808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚133489216870912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449536 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6542868646400 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89471993 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6542868646400 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚127121831691456 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89471993 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚682 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚140032337201664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 151722020 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37138052 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62276396 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚133646145627569 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151722020 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62276398 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚148097519171072 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0»¦à×%äM REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23643958 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 151748391 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚148097831620608 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6524559956761 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 $ _timestamp‚1.788467808277336e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚140032337201664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6524330975473 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60782010 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚141441489483264 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89445624 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6543102327808 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89445624 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62276398 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151748391 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288509440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚3861.463862418 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚133489216870912 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6543102327808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚133489216870912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449536 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6542868646400 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89471993 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6542868646400 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚127121831691456 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89471993 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚682 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚140032337201664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 151722020 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37138052 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62276396 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚133646145627569 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151722020 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62276398 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚148097519171072 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62276396 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23644638 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37138054 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚141441465543168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6656342137344 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23983957 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63171873 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6639353995264 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90783227 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90756884 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚135414232683520 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚150238327310336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚142053604681728 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚150238010653184 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6620310526995 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63171871 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6620539508075 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚135414232683520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚142053604681728 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90756884 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37667352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚706 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚135581112692867 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚142053353010688 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚135414250686464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚708 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚143484004015616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63171873 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273(Ì\á'®ÛO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚706 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚135581112692867 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚142053353010688 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚135414250686464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚708 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚143484004015616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63171873 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6639120327168 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61651309 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316657152 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _step‚271 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23983957 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63171873 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6639353995264 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90783227 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90756884 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚135414232683520 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚150238327310336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚142053604681728 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚150238010653184 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6620310526995 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63171871 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6620539508075 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚135414232683520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚142053604681728 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90756884 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37667352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0‹mR´2¯- _wandbruntime‚3917  _runtime‚3917ºU€ °àM H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6756310997504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153973722 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154000093 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚150281692825088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7884678641989043e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63190146 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚135453518720512 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚142094603073536 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217984 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _step‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37678152 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚128998332523296 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚142094854757888 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61669163 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚128998349562656 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90783576 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6641318034432 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23990331 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61668483 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚3349 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚142094854757888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153973722 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63190148 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚135453536723456 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚150281998983168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306158080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚135453536723456 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6641084353024 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚135453518720512 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90783576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚143525664045568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37678154 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚135620847559925 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6622269015981 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚142094603073536 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217984 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _step‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37678152 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚128998332523296 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚142094854757888 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61669163 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚128998349562656 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90783576 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6641318034432 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23990331 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61668483 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚3349 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚142094854757888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153973722 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63190148 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚135453536723456 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚150281998983168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306158080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚135453536723456 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6641084353024 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚135453518720512 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90783576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚143525664045568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37678154 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚135620847559925 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6622269015981 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154000093 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚150281692825088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7884678641989043e+09tE¡R2²- _wandbruntime‚3917  _runtime‚3917£_8Y³Ó 3loss/FBt1/assignment_loss‚5.801501780903588e-06 0loss/FRt1/assignment_loss‚3.1420245445691624 2loss/SRqqt4/assignment_loss‚1.9376059824770147 /loss/FRt2/detection_loss‚2.1643585688994653 &loss/total_loss‚46.380712660572144 epoch‚2 _runtime‚3917.386650336 0loss/FRt3/assignment_loss‚1.3893619535430786 1loss/FBt4/assignment_loss‚0.15362593285559947 /loss/FRt2/assignment_loss‚3.646187683321395 0loss/FRt4/assignment_loss‚3.4928197413682938 2loss/SRqqt1/assignment_loss‚1.6958784782327712 .loss/FBt1/detection_loss‚4.341832280158997 /loss/FBt2/detection_loss‚2.8720237639817325 2loss/SRqqt3/assignment_loss‚1.1196429065118234 /loss/FBt2/assignment_loss‚1.075291386193846 1loss/FBt3/assignment_loss‚0.13551962368244486 .loss/FRt1/detection_loss‚2.671702502629696 /loss/FRt3/detection_loss‚1.5001586006894227 .loss/FRt4/detection_loss‚2.734495861662759 0loss/SRqqt4/detection_loss‚2.505738038005251 /loss/FBt3/detection_loss‚0.7351505812789713 % _timestamp‚1.7884678642019012e+09 1loss/SRqqt2/assignment_loss‚1.576166076416319 1loss/SRqqt1/detection_loss‚1.9771674598256748 _step‚273 trainer/global_step‚3349 .loss/SRqqt2/detection_loss‚2.5507592193105 1loss/SRqqt3/detection_loss‚0.9434070108635795 -loss/FBt4/detection_loss‚2.01978866259257¦6 Ê^´Ø /loss/FRt3/detection_loss‚1.5001586006894227 .loss/FRt4/detection_loss‚2.734495861662759 0loss/SRqqt4/detection_loss‚2.505738038005251 /loss/FBt3/detection_loss‚0.7351505812789713 % _timestamp‚1.7884678642019012e+09 1loss/SRqqt2/assignment_loss‚1.576166076416319 1loss/SRqqt1/detection_loss‚1.9771674598256748 _step‚273 trainer/global_step‚3349 .loss/SRqqt2/detection_loss‚2.5507592193105 1loss/SRqqt3/detection_loss‚0.9434070108635795 -loss/FBt4/detection_loss‚2.01978866259257 3loss/FBt1/assignment_loss‚5.801501780903588e-06 0loss/FRt1/assignment_loss‚3.1420245445691624 2loss/SRqqt4/assignment_loss‚1.9376059824770147 /loss/FRt2/detection_loss‚2.1643585688994653 &loss/total_loss‚46.380712660572144 epoch‚2 _runtime‚3917.386650336 0loss/FRt3/assignment_loss‚1.3893619535430786 1loss/FBt4/assignment_loss‚0.15362593285559947 /loss/FRt2/assignment_loss‚3.646187683321395 0loss/FRt4/assignment_loss‚3.4928197413682938 2loss/SRqqt1/assignment_loss‚1.6958784782327712 .loss/FBt1/detection_loss‚4.341832280158997 /loss/FBt2/detection_loss‚2.8720237639817325 2loss/SRqqt3/assignment_loss‚1.1196429065118234 /loss/FBt2/assignment_loss‚1.075291386193846 1loss/FBt3/assignment_loss‚0.13551962368244486 .loss/FRt1/detection_loss‚2.671702502629696‘º÷8›’µ‚(:‡ ¤µçÔðÏÔ‡ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚137.693' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.42325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ub *s¶‚(:è ¤µçÔÀü¼ˆ disk.sdb.in‚ 3.49609375 disk.sdb.out‚425.90478515625 disk.dm-0.out‚ 50904.796875 disk.sda1.in‚0 disk.md0p1.out‚ 410.95703125 disk.rbd2.in‚0 disk.sda.in‚3.625 disk.sda.out‚425.90478515625" disk.nvme0n1.out‚ 15736.3046875! disk.sda2.out‚425.90478515625 disk.sdb2.in‚ 3.49609375! disk.sdb2.out‚425.90478515625) disk./.usagePercent‚83.15858441507022 disk.rbd2.out‚1.0625 disk.sdb1.in‚0" proc.memory.rssMB‚ 2953.3203125 disk.rbd0.out‚ 0.837890625" disk.nvme0n1.in‚328245.5234375 disk.sda1.out‚0 cpu‚25.10595122151483 proc.cpu.threads‚17 network.sent‚ 7.374614e+06 disk.md0.in‚ 7.12109375 disk.sda2.in‚3.625" disk.nvme1n1.in‚200282.4921875 disk.dm-0.in‚ 528528.015625 disk.md0p1.in‚ 7.12109375$ memory_percent‚32.41095542907715) proc.memory.availableMB‚ 11073.7890625* proc.memory.percent‚18.025636672973633 network.recv‚ 1.337562e+06$ disk./.usageGB‚2974.153778076172 disk.sdb1.out‚0 disk.md0.out‚ 410.95703125 disk.rbd0.in‚0.015625" disk.nvme1n1.out‚ 35168.4921875`¹r·‚(:… ³µçÔ؈އ gpu.0.gpu‚5.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚138.642' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.6605 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢Ä†>n¸‚(:ã ³µçԈ嗉 cpu‚25.105605494435043 disk.md0p1.in‚ 7.12109375 disk.sdb2.in‚ 3.49609375 disk.rbd2.in‚0" disk.nvme1n1.in‚200282.4921875 disk.sdb.out‚428.10986328125 disk.sdb1.in‚0 network.sent‚ 7.470523e+06 disk.sda.in‚3.625% proc.memory.availableMB‚ 11073.875 network.recv‚ 1.343522e+06! disk.sda2.out‚428.10986328125 disk.sdb.in‚ 3.49609375 disk.sdb1.out‚0" disk.nvme0n1.in‚328245.5234375# disk.nvme0n1.out‚15846.88671875 proc.cpu.threads‚17) disk./.usagePercent‚83.15788418867685 disk.sda1.in‚0 disk.md0.in‚ 7.12109375 disk.rbd2.out‚1.0625 disk.sda1.out‚0 disk.sda2.in‚3.625 disk.md0.out‚ 413.09375 disk.rbd0.out‚ 0.837890625" proc.memory.rssMB‚ 2953.3203125 disk.md0p1.out‚ 413.09375 disk.dm-0.in‚ 528528.015625$ disk./.usageGB‚2974.128734588623! disk.sdb2.out‚428.10986328125 disk.rbd0.in‚0.015625# disk.nvme1n1.out‚35168.77734375 disk.sda.out‚428.10986328125 disk.dm-0.out‚ 51015.6640625% memory_percent‚32.410430908203125* proc.memory.percent‚18.025636672973633C³d˹‚(:„ µçÔŀˇ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚86.502' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚21.6255 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N†urº‚(:ç µçÔÄ¢óˆ network.recv‚ 1.348231e+06) disk./.usagePercent‚83.17069059878712 disk.rbd2.out‚1.0625 proc.cpu.threads‚17! disk.sdb2.out‚429.45849609375 disk.sdb.in‚ 3.49609375 disk.dm-0.out‚51149.72265625 disk.sda1.in‚0 disk.sda2.in‚3.625 disk.md0p1.out‚ 414.390625# memory_percent‚32.4110746383667 network.sent‚ 7.476964e+06 disk.sda.in‚3.625 disk.rbd0.in‚0.015625* proc.memory.percent‚18.025636672973633" disk.nvme0n1.out‚ 15980.8359375 disk.md0.out‚ 414.390625" disk.nvme1n1.in‚200282.4921875 disk.sdb.out‚429.45849609375 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.dm-0.in‚ 528528.015625* proc.memory.availableMB‚11073.76953125 disk.sdb2.in‚ 3.49609375 disk.md0.in‚ 7.12109375" disk.nvme0n1.in‚328245.5234375 disk.sda1.out‚0! disk.sda2.out‚429.45849609375# disk.nvme1n1.out‚35168.88671875 disk.md0p1.in‚ 7.12109375 disk.rbd0.out‚ 0.837890625" proc.memory.rssMB‚ 2953.3203125$ disk./.usageGB‚2974.586753845215 disk.sda.out‚429.45849609375 disk.rbd2.in‚0 cpu‚25.10514989288003yŠù•2»- _wandbruntime‚3973  _runtime‚3973J>´Å¼¿ trainer/global_step‚3399 # _timestamp‚1.78846792070894e+09 _runtime‚3973.893869598 _step‚274 & lr-AdamW/pg1‚0.0007447564137902622 & lr-AdamW/pg2‚0.0007447564137902622T2Ê½Ä & lr-AdamW/pg1‚0.0007447564137902622 & lr-AdamW/pg2‚0.0007447564137902622 trainer/global_step‚3399 # _timestamp‚1.78846792070894e+09 _runtime‚3973.893869598 _step‚274’åRJ2¾- _wandbruntime‚3973  _runtime‚3973䊬¦_¿ÖO E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 trainer/global_step‚3399 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292717056 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6737569701888 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚137555814625015 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚144116122237952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38207452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92121179 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62540638 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚144116122237952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚145568178577920 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6853925079552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚137378552536064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38207450 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚707 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24332481 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24333186 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92094836 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92121179 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64085621 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64085623 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64085621 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚137378534533120 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚152422420314624 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚130837320037072 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚275 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64085623 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚145568202518016 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚137555568604575 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _runtime‚3973.894963913 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6718248567503 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92094836 ^LDeviceStats­²$} Monitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6718477548583 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚152422103657472 % _timestamp‚1.7884679207099545e+09 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚130837337076432 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6854217796608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144115870566912 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6737336033792 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6737336033792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316657152 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24332481 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24333186 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92094836 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92121179 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64085621 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64085623 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64085621 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚137378534533120 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚152422420314624 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚130837320037072 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚275 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64085623 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚145568202518016 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚137555568604575 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _runtime‚3973.894963913 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6718248567503 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92094836 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6718477548583 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚152422103657472 % _timestamp‚1.7884679207099545e+09 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚130837337076432 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6854217796608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144115870566912 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6737336033792 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6737336033792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316657152 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6737569701888 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚137555814625015 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚144116122237952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38207452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92121179 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62540638 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚144116122237952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚145568178577920 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6853925079552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224“Á•"ó2Á- _wandbruntime‚3974  _runtime‚39742…04æ&ÂàM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚3974.834459255 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚144157372314112 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64103896 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚144157372314112 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38218252 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156225424 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚137417820570112 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217984 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6739533741056 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62558775 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6739533741056 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 % _timestamp‚1.7884679216488435e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306158080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚130874867433856 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156251795 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24340521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚130874850394496 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64103898 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚137417838573056 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3399 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚152465804633600 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚137417838573056 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚276 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚144157120629760 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚145609886488064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚145609862547968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6739300059648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62558099 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6855942085632 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6720436037777 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273²ÔªKg%ÃåM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 % _timestamp‚1.7884679216488435e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306158080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚130874867433856 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156251795 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24340521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚130874850394496 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64103898 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚137417838573056 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3399 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚152465804633600 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚137417838573056 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚276 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚144157120629760 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚145609886488064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚145609862547968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6739300059648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62558099 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6855942085632 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6720436037777 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚3974.834459255 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚144157372314112 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64103896 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚144157372314112 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38218252 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156225424 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚137417820570112 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217984 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 =7DeviceStatsMo±„nitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6739533741056 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62558775 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6739533741056 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309”] ‰,2Ä- _wandbruntime‚3974  _runtime‚3974þF:YÅÓ /loss/FRt1/detection_loss‚3.0015362305887816 0loss/FBt1/assignment_loss‚0.4376812954075061 1loss/FBt3/assignment_loss‚0.07467678072981174 trainer/global_step‚3399 0loss/SRqqt2/detection_loss‚2.264088112860918 %loss/total_loss‚44.48292447870713 % _timestamp‚1.7884679216515024e+09 /loss/FRt2/detection_loss‚2.0592034098749257 2loss/SRqqt3/assignment_loss‚0.9102446655013475 /loss/FBt3/detection_loss‚0.7636038864409362 /loss/FRt1/assignment_loss‚3.182635442963962 .loss/FRt3/detection_loss‚1.508083989080347 0loss/FBt2/assignment_loss‚0.3266580494786574 1loss/SRqqt3/detection_loss‚1.0357469042920204 1loss/SRqqt4/detection_loss‚2.4417707435786724 epoch‚2 _runtime‚3974.836192003 0loss/FRt2/assignment_loss‚3.7300356310241076 _step‚277 0loss/FBt4/assignment_loss‚0.3717189351486013 0loss/FRt4/assignment_loss‚3.7941469929435034 1loss/SRqqt1/assignment_loss‚1.512729005744824 2loss/SRqqt2/assignment_loss‚1.9307859488762915 2loss/SRqqt4/assignment_loss‚2.0327577084923782 0loss/FRt3/assignment_loss‚1.3310044144020705 /loss/FRt4/detection_loss‚3.1961118470538747 0loss/SRqqt1/detection_loss‚2.088059584299723 .loss/FBt1/detection_loss‚2.171777688539945 .loss/FBt2/detection_loss‚1.721953288597219 /loss/FBt4/detection_loss‚2.5959139227867127럴q^ÆØ 0loss/FRt2/assignment_loss‚3.7300356310241076 _step‚277 0loss/FBt4/assignment_loss‚0.3717189351486013 0loss/FRt4/assignment_loss‚3.7941469929435034 1loss/SRqqt1/assignment_loss‚1.512729005744824 2loss/SRqqt2/assignment_loss‚1.9307859488762915 2loss/SRqqt4/assignment_loss‚2.0327577084923782 0loss/FRt3/assignment_loss‚1.3310044144020705 /loss/FRt4/detection_loss‚3.1961118470538747 0loss/SRqqt1/detection_loss‚2.088059584299723 .loss/FBt1/detection_loss‚2.171777688539945 .loss/FBt2/detection_loss‚1.721953288597219 /loss/FBt4/detection_loss‚2.5959139227867127 /loss/FRt1/detection_loss‚3.0015362305887816 0loss/FBt1/assignment_loss‚0.4376812954075061 1loss/FBt3/assignment_loss‚0.07467678072981174 trainer/global_step‚3399 0loss/SRqqt2/detection_loss‚2.264088112860918 %loss/total_loss‚44.48292447870713 % _timestamp‚1.7884679216515024e+09 /loss/FRt2/detection_loss‚2.0592034098749257 2loss/SRqqt3/assignment_loss‚0.9102446655013475 /loss/FBt3/detection_loss‚0.7636038864409362 /loss/FRt1/assignment_loss‚3.182635442963962 .loss/FRt3/detection_loss‚1.508083989080347 0loss/FBt2/assignment_loss‚0.3266580494786574 1loss/SRqqt3/detection_loss‚1.0357469042920204 1loss/SRqqt4/detection_loss‚2.4417707435786724 epoch‚2 _runtime‚3974.836192003•*r’Ç‚(:‡ ѵçԻ夈 gpu.0.gpu‚14.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚123.435' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.85875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹üœüxÈ‚(:í ѵçÔßÙň) disk./.usagePercent‚83.17256451767595 disk.rbd2.in‚0 disk.sdb1.in‚0 disk.sda.out‚430.98486328125 disk.sda2.in‚3.625 disk.md0p1.out‚ 415.85546875! disk.sdb2.out‚430.98486328125 disk.md0.out‚ 415.85546875" disk.nvme1n1.in‚200282.4921875 disk.sda1.out‚0 cpu‚25.104823433630475 network.recv‚ 1.352458e+06% memory_percent‚32.414841651916504* proc.memory.percent‚18.025636672973633 disk.rbd2.out‚1.0625 disk.md0.in‚ 7.12109375 network.sent‚ 7.480708e+06# disk.nvme1n1.out‚35237.84765625 disk.sdb.in‚ 3.49609375 disk.sdb.out‚430.98486328125 disk.sdb1.out‚0 disk.sda.in‚3.625* proc.memory.availableMB‚11073.15234375" proc.memory.rssMB‚ 2953.3203125" disk.nvme0n1.in‚328245.5234375# disk.nvme0n1.out‚16247.18359375 disk.sda1.in‚0 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 0.837890625 proc.cpu.threads‚17% disk./.usageGB‚2974.6537742614746 disk.dm-0.in‚ 528528.015625! disk.sda2.out‚430.98486328125 disk.sdb2.in‚ 3.49609375 disk.dm-0.out‚ 51485.03125 disk.md0p1.in‚ 7.12109375©i“Ö’É‚(:‡ àµçÔîÍƇ gpu.0.gpu‚42.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚142.313' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.57825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹>WhtÊ‚(:é àµçÔžÇû‡) disk./.usagePercent‚83.17431300377902 disk.sdb2.in‚ 3.49609375 disk.sda.out‚432.47216796875 disk.md0.out‚ 417.28515625 disk.sdb.in‚ 3.49609375 disk.sdb.out‚432.47216796875 disk.dm-0.out‚ 51644.8125) proc.memory.availableMB‚ 11072.8515625* proc.memory.percent‚18.025636672973633# disk.nvme0n1.out‚16379.24609375 disk.sda2.in‚3.625 disk.md0.in‚ 7.12109375 disk.rbd2.in‚0 disk.rbd2.out‚1.0625# disk.nvme1n1.out‚35265.56640625 disk.sdb1.in‚0" disk.nvme0n1.in‚328245.5234375 network.recv‚ 1.359243e+06# disk./.usageGB‚2974.71630859375 disk.rbd0.in‚0.015625 disk.md0p1.out‚ 417.28515625" proc.memory.rssMB‚ 2953.3203125 cpu‚25.104502156480226 disk.dm-0.in‚ 528528.015625 network.sent‚ 7.588329e+06 disk.sdb1.out‚0 disk.sda.in‚3.625 disk.sda1.out‚0 disk.sda1.in‚0! disk.sda2.out‚432.47216796875! disk.sdb2.out‚432.47216796875" disk.nvme1n1.in‚200282.4921875 disk.rbd0.out‚ 0.837890625 disk.md0p1.in‚ 7.12109375% memory_percent‚32.416677474975586 proc.cpu.threads‚17çs^¨‘Ë‚(:† ïµçÔÁü‡ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚102.658' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.6645 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0σìÛxÌ‚(:í ïµçÔè׆‹' proc.memory.availableMB‚ 11073.96875 disk.rbd2.in‚0 disk.rbd2.out‚1.0625" disk.nvme1n1.in‚200282.4921875# disk.nvme0n1.in‚328245.59765625 disk.sda1.in‚0* proc.memory.percent‚18.025636672973633$ disk./.usageGB‚2974.904182434082 disk.sda.in‚3.625 disk.md0p1.out‚ 419.27734375) disk./.usagePercent‚83.17956603498602 disk.md0.in‚ 7.12109375 disk.dm-0.in‚528528.08984375 disk.dm-0.out‚51794.05859375 disk.sda1.out‚0 disk.md0p1.in‚ 7.12109375 disk.rbd0.out‚ 0.837890625 proc.cpu.threads‚17 network.recv‚ 1.362127e+06! disk.sdb2.out‚434.52490234375 disk.md0.out‚ 419.27734375# disk.nvme1n1.out‚35357.98828125! disk.sda2.out‚434.52490234375 disk.sdb1.in‚0 disk.sdb2.in‚ 3.49609375 disk.rbd0.in‚0.015625 disk.sdb.in‚ 3.49609375 disk.sdb.out‚434.52490234375 disk.sda.out‚434.52490234375 disk.sda2.in‚3.625" proc.memory.rssMB‚ 2953.3203125 cpu‚25.104136057119863 network.sent‚ 7.594894e+06 disk.sdb1.out‚0" disk.nvme0n1.out‚ 16446.8203125$ memory_percent‚32.40985870361328oíûšÍ‚(:„ þµçÔžÄÚ† gpu.0.gpu‚6.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚112.572' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.143 gpu.0.smClock‚1230 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a/‚èk΂(:à þµçÔÁ²‚ˆ disk.sdb2.in‚ 3.49609375 disk.sdb2.out‚436.2216796875 disk.sdb.in‚ 3.49609375 disk.sdb1.out‚0 cpu‚25.103789693566068 disk.sdb1.in‚0 network.sent‚ 7.599011e+06 network.recv‚ 1.366813e+06) disk./.usagePercent‚83.17955163581419 disk.sdb.out‚436.2216796875% memory_percent‚32.412099838256836) proc.memory.availableMB‚ 11073.6015625 disk.dm-0.in‚528528.08984375 disk.md0p1.out‚ 420.90625 disk.md0.out‚ 420.90625 disk.sda.in‚3.625" proc.memory.rssMB‚ 2953.3203125 disk.sda2.out‚436.2216796875 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.sda.out‚436.2216796875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚3.625 disk.md0p1.in‚ 7.12109375 disk.md0.in‚ 7.12109375 disk.rbd0.out‚ 0.837890625* proc.memory.percent‚18.025636672973633 proc.cpu.threads‚17# disk.nvme0n1.in‚328245.59765625# disk.nvme0n1.out‚16646.30078125 disk.dm-0.out‚52115.5# disk.nvme1n1.out‚35469.19921875$ disk./.usageGB‚2974.903667449951 disk.rbd2.out‚1.0625" disk.nvme1n1.in‚200282.4921875@*V 2Ï- _wandbruntime‚4031  _runtime‚4031¶éÇÐÁ _step‚278 & lr-AdamW/pg1‚0.0007412048471084476 & lr-AdamW/pg2‚0.0007412048471084476 trainer/global_step‚3449 % _timestamp‚1.7884679779274206e+09 _runtime‚4031.112592479;2PÌÑÆ _step‚278 & lr-AdamW/pg1‚0.0007412048471084476 & lr-AdamW/pg2‚0.0007412048471084476 trainer/global_step‚3449 % _timestamp‚1.7884679779274206e+09 _runtime‚4031.112592479–¿¥F2Ò- _wandbruntime‚4031  _runtime‚4031Rúy†Ü'ÓÖO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6835551740416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93432788 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚139530270536723 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93459131 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚146178639794176 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚132713837908272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64999373 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38747552 trainer/global_step‚3449 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6835551740416 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 158432159 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24679191 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚147652377080320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚146178639794176 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚147652401020416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328448 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6816415589091 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 _runtime‚4031.113907425 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64999371 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚139342836382720 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚689 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚146178388123136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚146178388123136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚154606531927552 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224ä4 á'ÔÛO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚154606531927552 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6835551740416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93432788 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚139530270536723 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93459131 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚146178639794176 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚132713837908272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64999373 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38747552 trainer/global_step‚3449 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6835551740416 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 158432159 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24679191 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚147652377080320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚146178639794176 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚147652401020416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328448 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6816415589091 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 _runtime‚4031.113907425 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64999371 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚139342836382720 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚689 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚146178388123136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚146178388123136—4ßoj2Õ- _wandbruntime‚4032  _runtime‚4032HåÌ ÖàM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65017646 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚146219889870336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚139382140422656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65017648 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚154649907315712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 158477126 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412288 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚132751385305056 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6818374078285 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65017646 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚139382140422656 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63444259 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3449 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93459480 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352384 GADeviceStatsMonitor.on_train_batch_end/oversize_segmentNÖŠØs.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65017648 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚139382122419712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63444944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚139569759383341 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6955846265344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚154650217668096 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _runtime‚4032.282381458 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93459480 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6837515766272 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93485849 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24686590 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚685 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚139382122419712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚146219889870336 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24685907 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚139569513362693 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚146219638185984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 % _timestamp‚1.7884679790956502e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158477126 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158503497 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532u™ô%ë&×åM ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65017646 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚146219889870336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚139382140422656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65017648 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚154649907315712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 158477126 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412288 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚132751385305056 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6818374078285 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65017646 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚139382140422656 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63444259 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3449 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93459480 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65017648 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚139382122419712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63444944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚139569759383341 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6955846265344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚154650217668096 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _runtime‚4032.282381458 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93459480 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6837515766272 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93485849 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24686590 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚685 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚139382122419712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚146219889870336 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24685907 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚139569513362693 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚146219638185984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 % _timestamp‚1.7884679790956502e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158477126 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158503497 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚148240905679360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚141504480427991 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚156790574167040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚148241157350400 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚134590372818832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64312442 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39287652 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚148241157350400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6933767447040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6933767447040 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6914353629599 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚141307138232320 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94797083 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94770740 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39287650 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6934001115136 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148240905679360 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 % _timestamp‚1.7884680334409122e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65913123 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25024792 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚149736599522816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64313159 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚134590355779472 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6934001115136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚715 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 160710206 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94797083 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚141307156235264 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 trainer/global_step‚3499 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65913121 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7053974644224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 _step‚283 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65913123 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 160683861 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280¨©=á'èÛO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6934001115136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚715 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 160710206 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94797083 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚141307156235264 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 trainer/global_step‚3499 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65913121 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7053974644224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 _step‚283 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65913123 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 160683861 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365696 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚148240905679360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚141504480427991 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚156790574167040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚148241157350400 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚134590372818832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64312442 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39287652 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚148241157350400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6933767447040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6933767447040 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6914353629599 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚141307138232320 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94797083 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94770740 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39287650 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6934001115136 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148240905679360 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 % _timestamp‚1.7884680334409122e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65913123 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25024792 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚149736599522816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64313159 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚134590355779472 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0›­22é- _wandbruntime‚4087  _runtime‚4087‘ì’æ&êàM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚148282155742208 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25031477 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306158080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160755199 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6935731472896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚141346424269312 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6935965154304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚149778259552768 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6935731472896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 160728828 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7884680344337184e+09 _step‚284 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65931398 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚156833945688064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217984 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94797432 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65931398 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39298452 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚141544215295049 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6935965154304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25032176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7055968353280 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚148282407426560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚699 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚141346424269312 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64329929 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7055686135296 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚141346442272256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚701 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309¯tM êëåM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25031477 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306158080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160755199 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6935731472896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚141346424269312 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6935965154304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚149778259552768 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6935731472896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 160728828 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7884680344337184e+09 _step‚284 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65931398 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚156833945688064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217984 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94797432 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65931398 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39298452 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚141544215295049 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 713~‚Ã4511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6935965154304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25032176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7055968353280 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚148282407426560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚699 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚141346424269312 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64329929 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7055686135296 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚141346442272256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚701 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚148282155742208 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582œ1‰™Ð2ì- _wandbruntime‚4087  _runtime‚4087¨PÑ`íÚ 0loss/FRt1/assignment_loss‚3.2395148421155997 1loss/FBt4/assignment_loss‚0.08927116093827357 /loss/FBt3/detection_loss‚0.6788424769479592 trainer/global_step‚3499 _step‚285 0loss/FRt4/assignment_loss‚3.1281009563084305 2loss/SRqqt1/assignment_loss‚1.5982600027776284 1loss/SRqqt3/detection_loss‚0.7907017456258044 1loss/SRqqt4/assignment_loss‚2.040139381923983 1loss/SRqqt1/detection_loss‚1.9340361713067344 2loss/FBt3/assignment_loss‚0.059689438347009405 /loss/FRt2/detection_loss‚2.0950755522159548 0loss/SRqqt2/detection_loss‚2.990522745464529 1loss/SRqqt4/detection_loss‚2.7161850246690933 _runtime‚4087.620966273 0loss/FRt3/assignment_loss‚1.4895175647735597 .loss/FBt2/detection_loss‚2.143314845286883 0loss/FRt2/assignment_loss‚3.0652482410271964 1loss/FBt1/assignment_loss‚0.13065808295050374 /loss/FRt1/detection_loss‚2.9165393710136414 .loss/FBt1/detection_loss‚2.429689864317576 2loss/SRqqt2/assignment_loss‚1.5205709577671118 2loss/FBt2/assignment_loss‚0.061070744619848065 %loss/total_loss‚44.03904206172854 epoch‚2 % _timestamp‚1.7884680344357564e+09 2loss/SRqqt3/assignment_loss‚1.2609217706930769 /loss/FRt3/detection_loss‚1.5391214104493458 .loss/FRt4/detection_loss‚3.020776123836123 /loss/FBt4/detection_loss‚3.1012735863526664”áO eîß 2loss/FBt3/assignment_loss‚0.059689438347009405 /loss/FRt2/detection_loss‚2.0950755522159548 0loss/SRqqt2/detection_loss‚2.990522745464529 1loss/SRqqt4/detection_loss‚2.7161850246690933 _runtime‚4087.620966273 0loss/FRt3/assignment_loss‚1.4895175647735597 .loss/FBt2/detection_loss‚2.143314845286883 0loss/FRt2/assignment_loss‚3.0652482410271964 1loss/FBt1/assignment_loss‚0.13065808295050374 /loss/FRt1/detection_loss‚2.9165393710136414 .loss/FBt1/detection_loss‚2.429689864317576 2loss/SRqqt2/assignment_loss‚1.5205709577671118 2loss/FBt2/assignment_loss‚0.061070744619848065 %loss/total_loss‚44.03904206172854 epoch‚2 % _timestamp‚1.7884680344357564e+09 2loss/SRqqt3/assignment_loss‚1.2609217706930769 /loss/FRt3/detection_loss‚1.5391214104493458 .loss/FRt4/detection_loss‚3.020776123836123 /loss/FBt4/detection_loss‚3.1012735863526664 0loss/FRt1/assignment_loss‚3.2395148421155997 1loss/FBt4/assignment_loss‚0.08927116093827357 /loss/FBt3/detection_loss‚0.6788424769479592 trainer/global_step‚3499 _step‚285 0loss/FRt4/assignment_loss‚3.1281009563084305 2loss/SRqqt1/assignment_loss‚1.5982600027776284 1loss/SRqqt3/detection_loss‚0.7907017456258044 1loss/SRqqt4/assignment_loss‚2.040139381923983 1loss/SRqqt1/detection_loss‚1.9340361713067344{'”æ‘ï‚(:† ɶçԢʳ‡ gpu.0.gpu‚7.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.357' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.83925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ú5DÅqð‚(:æ ɶçÔÀ¬ë‡ disk.md0.out‚ 428.74609375! disk.nvme1n1.out‚ 35836.671875* proc.memory.percent‚18.025636672973633# disk./.usageGB‚2974.65189743042 disk.dm-0.out‚53148.92578125 disk.sda2.out‚444.3759765625 disk.md0p1.out‚ 428.74609375 disk.sdb2.in‚ 3.49609375# disk.nvme1n1.in‚200374.74609375 disk.sda.out‚444.3759765625) disk./.usagePercent‚83.17251204069414 disk.sda1.in‚0 disk.sda2.in‚3.625 disk.md0p1.in‚ 7.12109375 disk.md0.in‚ 7.12109375 disk.rbd0.in‚0.015625 cpu‚25.10220948361978 proc.cpu.threads‚17 disk.rbd0.out‚ 0.837890625 disk.sda.in‚3.625" proc.memory.rssMB‚ 2953.3203125 network.recv‚ 1.390099e+06 disk.sdb1.in‚0 disk.dm-0.in‚528639.73046875 disk.rbd2.out‚1.0625 disk.sdb1.out‚0 disk.sda1.out‚0 disk.rbd2.in‚0 disk.sdb.in‚ 3.49609375 disk.sdb.out‚444.3759765625! disk.nvme0n1.in‚ 328264.984375# disk.nvme0n1.out‚17312.25390625$ memory_percent‚32.41434097290039( proc.memory.availableMB‚ 11073.234375 network.sent‚ 7.729726e+06 disk.sdb2.out‚444.37597656255,!Š’ñ‚(:‡ ضçÔ‘­å‡ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚137.693' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.42325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EO÷lò‚(:á ضçÔÛÙ›‰ disk.rbd0.in‚0.015625( disk./.usagePercent‚83.1725716639316# disk.nvme0n1.in‚328264.99609375 disk.dm-0.out‚ 53417.1328125 disk.sda1.in‚0 disk.md0p1.out‚ 430.1328125 disk.sda.out‚ 445.82421875* proc.memory.availableMB‚11073.48046875 disk.sdb1.in‚0% disk./.usageGB‚2974.6540298461914 disk.md0p1.in‚ 7.12109375 disk.md0.out‚ 430.1328125 disk.rbd2.out‚1.0625 network.recv‚ 1.396895e+06 disk.md0.in‚ 7.12109375 proc.cpu.threads‚17# disk.nvme0n1.out‚17578.20703125 disk.sda1.out‚0 disk.sda2.in‚3.625 disk.sda2.out‚ 445.82421875 disk.sdb2.in‚ 3.49609375# disk.nvme1n1.out‚35838.92578125 disk.sdb.in‚ 3.49609375 disk.sdb.out‚ 445.82421875 network.sent‚ 7.839359e+06 disk.dm-0.in‚528639.7421875 disk.rbd0.out‚ 0.837890625 disk.rbd2.in‚0 disk.sda.in‚3.625$ memory_percent‚32.41283893585205# disk.nvme1n1.in‚200374.74609375 disk.sdb1.out‚0" proc.memory.rssMB‚ 2953.3203125* proc.memory.percent‚18.025636672973633 cpu‚25.10188221550259 disk.sdb2.out‚ 445.824218757Ü-Ê’ó‚(:‡ ç¶çÔý©¼‡ gpu.0.gpu‚27.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚141.087' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.27175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­%uô‚(:ê ç¶çÔ“ô‡ cpu‚25.101531182076062 disk.sdb.out‚447.19091796875 disk.dm-0.out‚ 53653.109375 disk.md0p1.in‚ 7.12109375# disk.nvme1n1.in‚200374.74609375" disk.nvme0n1.out‚ 17711.7421875 disk.md0p1.out‚ 431.44921875! disk.sdb2.out‚447.19091796875" proc.memory.rssMB‚ 2953.3203125 network.recv‚ 1.40069e+06% disk./.usageGB‚2974.7789192199707 disk.sdb1.in‚0 disk.sda.out‚447.19091796875 disk.sda2.in‚3.625 disk.sdb2.in‚ 3.49609375" disk.nvme1n1.out‚ 35941.3671875 network.sent‚ 7.844654e+06) disk./.usagePercent‚83.17606362309274 disk.sdb1.out‚0 disk.sda.in‚3.625! disk.sda2.out‚447.19091796875' proc.memory.availableMB‚ 11073.46875 disk.dm-0.in‚528639.7421875 disk.md0.in‚ 7.12109375 disk.sdb.in‚ 3.49609375# disk.nvme0n1.in‚328264.99609375 disk.sda1.in‚0 disk.rbd2.out‚1.0625 disk.sda1.out‚0 disk.md0.out‚ 431.44921875 disk.rbd0.in‚0.015625 disk.rbd2.in‚0$ memory_percent‚32.41291046142578* proc.memory.percent‚18.025636672973633 proc.cpu.threads‚17 disk.rbd0.out‚ 0.837890625%µã-õ‚(:… ö¶çԦ™‡ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚139.868' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.967 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿W+…fö‚(:Û ö¶çÔ¶î‡ network.sent‚ 7.848729e+06 disk.rbd2.out‚1.0625" proc.memory.rssMB‚ 2953.3203125 cpu‚25.10116680587394) disk./.usagePercent‚83.15000304195837$ disk./.usageGB‚2973.846866607666 disk.rbd0.in‚0.015625 disk.sdb.in‚ 3.49609375 disk.sda.in‚3.625 disk.sda2.out‚448.9833984375 disk.md0p1.out‚ 433.1796875 disk.md0.out‚ 433.1796875 disk.sdb2.in‚ 3.49609375 network.recv‚ 1.405075e+06 disk.rbd0.out‚ 0.837890625# disk.nvme0n1.in‚328264.99609375 disk.dm-0.out‚ 53861.328125$ memory_percent‚32.41443634033203 disk.sdb.out‚448.9833984375 disk.sda.out‚448.9833984375! disk.nvme0n1.out‚ 17871.859375 disk.md0.in‚ 7.12109375 disk.sdb1.in‚0 disk.sda2.in‚3.625 disk.nvme1n1.in‚ 200374.75 disk.sda1.in‚0* proc.memory.percent‚18.025636672973633 disk.sdb2.out‚448.9833984375 disk.nvme1n1.out‚ 35989.46875 disk.sdb1.out‚0 disk.dm-0.in‚528639.74609375' proc.memory.availableMB‚ 11073.21875 disk.rbd2.in‚0 disk.sda1.out‚0 disk.md0p1.in‚ 7.12109375 proc.cpu.threads‚17 0,Q2÷- _wandbruntime‚4140  _runtime‚4140+}@@ÇøÁ trainer/global_step‚3549 % _timestamp‚1.7884680869621563e+09 _runtime‚4140.146962665 _step‚286 & lr-AdamW/pg1‚0.0007338967922970386 & lr-AdamW/pg2‚0.00073389679229703868ÒÝ0ÌùÆ & lr-AdamW/pg1‚0.0007338967922970386 & lr-AdamW/pg2‚0.0007338967922970386 trainer/global_step‚3549 % _timestamp‚1.7884680869621563e+09 _runtime‚4140.146962665 _step‚286ž¢Ä`/2ú- _wandbruntime‚4140  _runtime‚4140§º3Ü'ûÖO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚702 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66826873 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7012062689027 trainer/global_step‚3549 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚151820774085120 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96108692 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25369029 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚143271440081920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66826871 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39827752 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7884680869628706e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚150303423235584 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65197481 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7012291670107 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚150303423235584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚158974440958976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚136466890690032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚4140.147881329 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 162935563 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚150303674906624 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚143478936339699 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚143479182360139 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96135035 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65196779 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚150303674906624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25369729 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7032216821760 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚151820774085120 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96108692 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25369029 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚143271440081920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66826871 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39827752 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7884680869628706e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚150303423235584 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65197481 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7012291670107 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚150303423235584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚158974440958976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚136466890690032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚4140.147881329 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 162935563 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚150303674906624 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚143478936339699 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚143479182360139 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96135035 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65196779 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚150303674906624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25369729 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7032216821760 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163006901 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚143518671206757 C;Device%!ÁðJ$StatsMonitor.on_train_batch_end/segment.all.allocated‚582 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162980530 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96161753 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 $ _timestamp‚1.788468087919061e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚150344673298432 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7155952545792 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65214647 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚650 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66845148 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7155657744896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7014250159301 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚150344924982784 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25376745 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 162980530 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 trainer/global_step‚3549 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚143310744121856 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96161753 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39838554 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚151862458055168 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚159018434541056 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66845146 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚151862481995264 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66845148 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 _step‚288 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7034180860928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25376095 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65215299 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚143518425186109 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7033947179520 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7034180860928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7033947179520 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 _runtime‚ 4141.1045713 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163006901 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66845146 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚143310726118912 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96135384 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚136504404008096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚652 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7014021178013 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚150344924982784Ò0qØè&ÿâM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66845146 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚151862481995264 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66845148 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 _step‚288 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7034180860928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25376095 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65215299 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚143518425186109 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7033947179520 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7034180860928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7033947179520 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 _runtime‚ 4141.1045713 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163006901 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66845146 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚143310726118912 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96135384 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚136504404008096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚652 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7014021178013 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚150344924982784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800896 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚143310744121856 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163006901 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚143518671206757 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162980530 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96161753 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 $ _timestamp‚1.788468087919061e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚150344673298432 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7155952545792 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65214647 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚650 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66845148 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7155657744896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7014250159301 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚150344924982784 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25376745 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 162980530 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 trainer/global_step‚3549 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚143310744121856 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96161753 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39838554 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚151862458055168 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚159018434541056 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 &J2€- _wandbruntime‚4141  _runtime‚4141š`WÑ 0loss/FRt3/assignment_loss‚1.6819421595601893 /loss/FRt4/assignment_loss‚3.401820292075475 1loss/SRqqt3/assignment_loss‚1.150395697684257 .loss/FRt1/detection_loss‚2.631595647648761 1loss/SRqqt1/detection_loss‚1.8251910267806635 % _timestamp‚1.7884680879211595e+09 2loss/SRqqt1/assignment_loss‚1.6119373524334373 epoch‚2 _runtime‚4141.105842785 2loss/SRqqt4/assignment_loss‚1.8791380632254813 0loss/FBt2/assignment_loss‚0.1693739052877979 /loss/FRt2/assignment_loss‚4.373962647385067 /loss/FBt1/assignment_loss‚0.283696731775611 .loss/FRt4/detection_loss‚2.814389818906784 1loss/SRqqt4/detection_loss‚2.3787766132089825 0loss/FRt1/assignment_loss‚3.2924819102412775 1loss/SRqqt2/assignment_loss‚1.335628467767189 1loss/FBt3/assignment_loss‚0.10350005037431767 /loss/FBt2/detection_loss‚1.8674703696194817 0loss/FBt4/assignment_loss‚0.3229425800964236 1loss/SRqqt3/detection_loss‚1.1206348415512544 _step‚289 0loss/SRqqt2/detection_loss‚2.558911442756653 .loss/FBt1/detection_loss‚3.089339941740036 .loss/FBt3/detection_loss‚0.568467157486249 trainer/global_step‚3549 /loss/FRt2/detection_loss‚1.9336937103912026 /loss/FRt3/detection_loss‚1.6814564967444583 /loss/FBt4/detection_loss‚2.5289909869432448 &loss/total_loss‚44.605737911684294›u”\‚Ö 0loss/SRqqt2/detection_loss‚2.558911442756653 .loss/FBt1/detection_loss‚3.089339941740036 .loss/FBt3/detection_loss‚0.568467157486249 trainer/global_step‚3549 /loss/FRt2/detection_loss‚1.9336937103912026 /loss/FRt3/detection_loss‚1.6814564967444583 /loss/FBt4/detection_loss‚2.5289909869432448 &loss/total_loss‚44.605737911684294 0loss/FRt3/assignment_loss‚1.6819421595601893 /loss/FRt4/assignment_loss‚3.401820292075475 1loss/SRqqt3/assignment_loss‚1.150395697684257 .loss/FRt1/detection_loss‚2.631595647648761 1loss/SRqqt1/detection_loss‚1.8251910267806635 % _timestamp‚1.7884680879211595e+09 2loss/SRqqt1/assignment_loss‚1.6119373524334373 epoch‚2 _runtime‚4141.105842785 2loss/SRqqt4/assignment_loss‚1.8791380632254813 0loss/FBt2/assignment_loss‚0.1693739052877979 /loss/FRt2/assignment_loss‚4.373962647385067 /loss/FBt1/assignment_loss‚0.283696731775611 .loss/FRt4/detection_loss‚2.814389818906784 1loss/SRqqt4/detection_loss‚2.3787766132089825 0loss/FRt1/assignment_loss‚3.2924819102412775 1loss/SRqqt2/assignment_loss‚1.335628467767189 1loss/FBt3/assignment_loss‚0.10350005037431767 /loss/FBt2/detection_loss‚1.8674703696194817 0loss/FBt4/assignment_loss‚0.3229425800964236 1loss/SRqqt3/detection_loss‚1.1206348415512544 _step‚289¡é©é!‘ƒ‚(:† …·çÔ´áõ‡ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚134.298' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.5745 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0T^|¢m„‚(:â …·çÔ”ªÌˆ disk.sda2.in‚3.625" proc.memory.rssMB‚ 2953.3203125 network.recv‚ 1.408741e+06% disk./.usageGB‚2973.8483505249023# disk.nvme0n1.in‚328264.99609375" disk.nvme0n1.out‚ 17984.1640625 disk.md0p1.out‚ 434.8515625 disk.rbd0.out‚ 0.837890625$ memory_percent‚32.41307735443115 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sdb2.in‚ 3.49609375 disk.rbd2.in‚0* proc.memory.availableMB‚11073.44140625 disk.sdb1.out‚0 disk.sda.out‚ 450.732421875 disk.sda1.out‚0 disk.md0p1.in‚ 7.12109375 disk.md0.out‚ 434.8515625* proc.memory.percent‚18.025636672973633 cpu‚25.100675911542588) disk./.usagePercent‚83.15004453290538 disk.sdb.out‚ 450.732421875 disk.sdb2.out‚ 450.732421875 disk.sdb1.in‚0 disk.md0.in‚ 7.12109375 disk.rbd0.in‚0.015625 disk.sda.in‚3.625 network.sent‚ 7.853727e+06 disk.rbd2.out‚1.0625 disk.nvme1n1.in‚ 200374.75# disk.nvme1n1.out‚36091.59765625 disk.sdb.in‚ 3.49609375 disk.dm-0.out‚54075.76171875 disk.dm-0.in‚528639.74609375 disk.sda2.out‚ 450.732421875Aág=h…‚(:Ý ”·çÔÒ὇% disk./.usageGB‚2973.8483657836914 disk.md0.in‚ 7.12109375 disk.md0.out‚ 436.3984375 disk.sdb.out‚ 452.341796875# disk.nvme0n1.in‚328264.99609375 disk.dm-0.in‚528639.74609375 disk.md0p1.in‚ 7.12109375 disk.md0p1.out‚ 436.3984375 disk.sda1.out‚0( disk./.usagePercent‚83.1500449595475 disk.sdb2.out‚ 452.341796875 disk.sdb.in‚ 3.49609375 network.recv‚ 1.415533e+06! disk.nvme0n1.out‚ 17984.234375 disk.dm-0.out‚ 54200.1796875 disk.sda2.in‚3.625 disk.sda2.out‚ 452.341796875 disk.rbd0.in‚0.015625 disk.sdb1.in‚0 disk.sda.in‚3.625$ memory_percent‚32.41314888000488 cpu‚25.100099357960982 network.sent‚ 7.961327e+06 disk.rbd0.out‚ 0.837890625 disk.rbd2.in‚0 disk.sda.out‚ 452.341796875) proc.memory.availableMB‚ 11073.4296875" proc.memory.rssMB‚ 2953.3203125 proc.cpu.threads‚17 disk.sdb2.in‚ 3.49609375 disk.rbd2.out‚1.0625 disk.nvme1n1.in‚ 200374.75* proc.memory.percent‚18.025636672973633" disk.nvme1n1.out‚ 36215.9453125 disk.sdb1.out‚0 disk.sda1.in‚0ZbE‡†‚(:… ”·çÔ¬¾µ‡ gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚142.036' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.509 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0']ÿ*œ‡‚(:‘ £·çԻט† gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚119.368' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.841999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çw\sˆ‚(:è £·çÔ¾âÚ† disk.nvme1n1.in‚ 200374.75 disk.sdb.in‚ 3.49609375 disk.sdb.out‚453.7333984375 network.sent‚ 7.966658e+06 disk.sdb1.out‚0 disk.sda.in‚3.625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚453.7333984375 disk.md0p1.out‚ 437.74609375 disk.rbd0.in‚0.015625 network.recv‚ 1.419449e+06) disk./.usagePercent‚83.14675426881048 disk.sda.out‚453.7333984375 disk.dm-0.in‚528639.74609375 disk.sdb2.out‚453.7333984375" proc.memory.rssMB‚ 2953.3203125 cpu‚25.099746360293683 proc.cpu.threads‚17# disk.nvme0n1.in‚328264.99609375# disk.nvme0n1.out‚17986.00390625 disk.dm-0.out‚ 54309.8359375# disk.nvme1n1.out‚36323.83203125 disk.sda2.in‚3.625 disk.md0.out‚ 437.74609375 disk.rbd2.out‚1.0625* proc.memory.availableMB‚11073.42578125* proc.memory.percent‚18.025636672973633 disk.md0p1.in‚ 7.12109375 disk.sdb1.in‚0 disk.sdb2.in‚ 3.49609375 disk.md0.in‚ 7.12109375$ memory_percent‚32.41317272186279% disk./.usageGB‚2973.7306747436523 disk.rbd0.out‚ 0.837890625 disk.rbd2.in‚0E¡*2‰- _wandbruntime‚4192  _runtime‚4192*ŽþÆŠÀ & lr-AdamW/pg1‚0.0007301418271551133 & lr-AdamW/pg2‚0.0007301418271551133 trainer/global_step‚3599 $ _timestamp‚1.788468138978973e+09 _runtime‚4192.163857043 _step‚290@Ï5XË‹Å _step‚290 & lr-AdamW/pg1‚0.0007301418271551133 & lr-AdamW/pg2‚0.0007301418271551133 trainer/global_step‚3599 $ _timestamp‚1.788468138978973e+09 _runtime‚4192.163857043¢Ù8Ã2Œ- _wandbruntime‚4192  _runtime‚4192>£ŠÖO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚145235759934464 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚161158481750528 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚153904972587520 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152365940791808 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165213610 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7130432528384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97472987 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619904 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7130198860288 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚161158796310528 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7253799782912 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165187265 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25718467 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚145453392251407 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97472987 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚145235759934464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25717744 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚152366192462848 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7130198860288 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40367852 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚723 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _step‚291K¬7©á'ŽÛO C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67740621 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884681389797177e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40367850 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚152365940791808 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚145235741931520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165187265 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25718467 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚145453392251407 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97472987 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚145235759934464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25717744 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚152366192462848 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7130198860288 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40367852 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚723 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _step‚291 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚145235759934464 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚161158481750528 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚153904972587520 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152365940791808 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165213610 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7130432528384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97472987 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619904 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7130198860288 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚161158796310528 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7253799782912 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 B:üÓæ&àM B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚138380938918656 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _runtime‚4193.124087032 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165258603 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412288 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚152407442539008 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚153946656557568 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66103020 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚138380921879296 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165258603 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97473336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7255789027840 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67758898 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚152407442539008 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25724368 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7255502615552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7111959218521 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67758898 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97499705 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7132162886144 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚145493127118465 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 % _timestamp‚1.7884681399382937e+09 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67758896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚145492881097817œLñ߇‘åM PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67758896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚145492881097817 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚138380938918656 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _runtime‚4193.124087032 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165258603 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412288 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚152407442539008 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚153946656557568 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66103020 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚138380921879296 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165258603 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97473336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7255789027840 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67758898 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚152407442539008 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25724368 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.alloKѺdcated‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7255502615552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7111959218521 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67758898 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97499705 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7132162886144 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚145493127118465 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 % _timestamp‚1.7884681399382937e+09 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248¤‚l‹}2’- _wandbruntime‚4193  _runtime‚4193ØŠŽFZ“Ô 2loss/SRqqt3/assignment_loss‚1.0289467786585813 1loss/FBt2/assignment_loss‚0.12723015473416344 /loss/FRt3/detection_loss‚1.4417805463075637 1loss/SRqqt2/detection_loss‚2.4335945043712854 2loss/SRqqt1/assignment_loss‚1.4638011349737645 2loss/SRqqt2/assignment_loss‚1.6685755716171116 /loss/FRt4/assignment_loss‚3.765244866410891 1loss/FBt3/assignment_loss‚0.07268715276181525 /loss/FRt4/detection_loss‚3.1750884130597115 0loss/SRqqt1/detection_loss‚2.721893520355225 1loss/SRqqt4/detection_loss‚2.4574348494924347 % _timestamp‚1.7884681399414067e+09 /loss/FRt2/assignment_loss‚3.865118298265669 .loss/FRt1/detection_loss‚2.796829338996641 .loss/FBt1/detection_loss‚2.280619672366551 epoch‚2 0loss/FBt3/detection_loss‚0.49964079800267497 /loss/FRt1/assignment_loss‚3.693760960332809 0loss/FBt4/assignment_loss‚1.0439688364612418 1loss/SRqqt3/detection_loss‚0.9667056208378391 _runtime‚4193.126147791 /loss/FBt4/detection_loss‚3.1759785612424216 .loss/FBt2/detection_loss‚2.669520588148208 %loss/total_loss‚47.61034659355174 _step‚293 0loss/SRqqt4/assignment_loss‚2.09228620035895 1loss/FBt1/assignment_loss‚0.29650987304341925 /loss/FRt2/detection_loss‚2.4090003363364505 trainer/global_step‚3599 0loss/FRt3/assignment_loss‚1.4641300164163114l‰µ—_”Ù 0loss/SRqqt4/assignment_loss‚2.09228620035895 1loss/FBt1/assignment_loss‚0.29650987304341925 /loss/FRt2/detection_loss‚2.4090003363364505 trainer/global_step‚3599 0loss/FRt3/assignment_loss‚1.4641300164163114 2loss/SRqqt3/assignment_loss‚1.0289467786585813 1loss/FBt2/assignment_loss‚0.12723015473416344 /loss/FRt3/detection_loss‚1.4417805463075637 1loss/SRqqt2/detection_loss‚2.4335945043712854 2loss/SRqqt1/assignment_loss‚1.4638011349737645 2loss/SRqqt2/assignment_loss‚1.6685755716171116 /loss/FRt4/assignment_loss‚3.765244866410891 1loss/FBt3/assignment_loss‚0.07268715276181525 /loss/FRt4/detection_loss‚3.1750884130597115 0loss/SRqqt1/detection_loss‚2.721893520355225 1loss/SRqqt4/detection_loss‚2.4574348494924347 % _timestamp‚1.7884681399414067e+09 /loss/FRt2/assignment_loss‚3.865118298265669 .loss/FRt1/detection_loss‚2.796829338996641 .loss/FBt1/detection_loss‚2.280619672366551 epoch‚2 0loss/FBt3/detection_loss‚0.49964079800267497 /loss/FRt1/assignment_loss‚3.693760960332809 0loss/FBt4/assignment_loss‚1.0439688364612418 1loss/SRqqt3/detection_loss‚0.9667056208378391 _runtime‚4193.126147791 /loss/FBt4/detection_loss‚3.1759785612424216 .loss/FBt2/detection_loss‚2.669520588148208 %loss/total_loss‚47.61034659355174 _step‚293¥¾¦á’•‚(:‡ ²·çÔ“„Õ† gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚142.313' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.57825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ8'Lf–‚(:Û ²·çÔ©±Š‡ disk.sdb2.in‚ 3.49609375 disk.sda.in‚3.625 disk.md0p1.out‚ 439.8515625 cpu‚25.09932925724007$ disk./.usageGB‚2973.458080291748 disk.sdb2.out‚ 455.8984375 disk.rbd0.out‚ 0.837890625 disk.rbd2.in‚0* proc.memory.percent‚18.025636672973633 proc.cpu.threads‚17 disk.rbd0.in‚0.015625 disk.sdb1.in‚0 disk.sda1.in‚0 disk.md0p1.in‚ 7.12109375 disk.rbd2.out‚1.0625# disk.nvme1n1.out‚36366.80859375 disk.sdb1.out‚0 network.recv‚ 1.424073e+06" disk.nvme1n1.in‚200383.3671875# disk.nvme0n1.in‚328294.05078125 disk.dm-0.in‚528677.41796875 disk.sda2.in‚3.625 network.sent‚ 7.972997e+06 disk.sda1.out‚0% memory_percent‚32.416439056396484 disk.md0.in‚ 7.12109375 disk.sdb.out‚ 455.8984375" disk.nvme0n1.out‚ 17986.0703125( proc.memory.availableMB‚ 11072.890625" proc.memory.rssMB‚ 2953.3203125 disk.md0.out‚ 439.8515625 disk.sdb.in‚ 3.49609375 disk.sda.out‚ 455.8984375 disk.dm-0.out‚54352.87890625 disk.sda2.out‚ 455.8984375( disk./.usagePercent‚83.1391324138456ùú‘—‚(:† Á·çÔ©üè‡ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚133.626' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.4065 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+ãÍw˜‚(:ì Á·çÔ¸á÷ˆ% disk./.usageGB‚2973.5822677612305 disk.md0.in‚ 7.12109375 disk.sdb1.in‚0# disk.nvme0n1.in‚328294.05078125 disk.sdb2.in‚ 3.49609375 disk.rbd0.out‚ 0.837890625 cpu‚25.099014760716617" disk.nvme1n1.in‚200383.3671875) disk./.usagePercent‚83.14260474746882 disk.sda2.out‚457.1787109375 disk.md0p1.out‚ 441.08984375 disk.md0.out‚ 441.08984375% memory_percent‚32.413434982299805 disk.rbd2.in‚0 network.sent‚ 8.080239e+06 disk.sdb1.out‚0 disk.dm-0.out‚54439.83203125 disk.sda1.in‚0 disk.rbd0.in‚0.015625 disk.sdb.out‚457.1787109375 disk.md0p1.in‚ 7.12109375 proc.cpu.threads‚17 disk.sda.in‚3.625 disk.sda1.out‚0 disk.sdb2.out‚457.1787109375" proc.memory.rssMB‚ 2953.3203125* proc.memory.percent‚18.025636672973633 disk.rbd2.out‚1.0625 disk.sda.out‚457.1787109375! disk.nvme0n1.out‚ 17986.796875 disk.dm-0.in‚528677.41796875 disk.sda2.in‚3.625 disk.sdb.in‚ 3.49609375) proc.memory.availableMB‚ 11073.3828125# disk.nvme1n1.out‚36453.03515625 network.recv‚ 1.430991e+06cãÙÞ‘™‚(:† зçÔ«£·‡ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02–qš‚(:æ зçÔܦ߇ proc.cpu.threads‚17) disk./.usagePercent‚83.14452645027606 disk.sdb.in‚ 3.49609375 disk.sda.in‚3.625 disk.sda2.out‚ 458.748046875 disk.sdb2.in‚ 3.49609375) proc.memory.percent‚18.49656105041504 cpu‚25.106127906543467 disk.sda.out‚ 458.748046875# disk.nvme1n1.out‚36607.58984375 disk.dm-0.out‚ 54594.4453125 disk.sdb1.in‚0% disk./.usageGB‚2973.6509971618652 disk.rbd2.in‚0# disk.nvme0n1.in‚328304.30078125 disk.dm-0.in‚528695.69921875 disk.sda1.in‚0 disk.md0.in‚ 7.12109375 network.recv‚ 1.435449e+06 disk.sdb.out‚ 458.748046875 disk.sda1.out‚0 disk.md0p1.in‚ 7.12109375 disk.md0p1.out‚ 442.6015625 disk.sdb2.out‚ 458.748046875 disk.rbd0.in‚0.015625" proc.memory.rssMB‚ 3030.4765625 disk.rbd2.out‚1.0625" disk.nvme1n1.in‚200391.3984375 disk.sdb1.out‚0# disk.nvme0n1.out‚17986.85546875 disk.rbd0.out‚ 0.837890625$ memory_percent‚32.88724422454834 network.sent‚ 8.086208e+06 disk.sda2.in‚3.625 disk.md0.out‚ 442.6015625* proc.memory.availableMB‚10995.75390625qÖ»Bœ›‚(:‘ ß·çÔÛÉÕ… gpu.0.gpu‚46.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªšdÊrœ‚(:ç ß·çÔ¾ù˜‡ disk.md0.in‚ 7.12109375 network.recv‚ 1.439193e+06) proc.memory.percent‚18.49656105041504 cpu‚25.119436098716157 disk.nvme1n1.out‚36691.75 disk.sdb.out‚460.97802734375 disk.sda.in‚3.625 disk.dm-0.in‚528695.69921875 disk.sda2.in‚3.625 disk.sdb.in‚ 3.49609375 disk.sda.out‚460.97802734375 disk.md0p1.out‚ 444.75390625 proc.cpu.threads‚17" disk.nvme1n1.in‚200391.3984375% memory_percent‚32.887935638427734( disk./.usagePercent‚83.1481695474112# disk./.usageGB‚2973.78129196167 disk.sdb1.in‚0# disk.nvme0n1.out‚17987.18359375 disk.dm-0.out‚54678.93359375 disk.sda1.in‚0 disk.sda1.out‚0! disk.sdb2.out‚460.97802734375 network.sent‚ 8.093678e+06 disk.sdb2.in‚ 3.49609375 disk.rbd0.out‚ 0.837890625 disk.rbd2.in‚0! disk.sda2.out‚460.97802734375 disk.rbd0.in‚0.015625( proc.memory.availableMB‚ 10995.640625 disk.rbd2.out‚1.0625 disk.sdb1.out‚0# disk.nvme0n1.in‚328304.30078125 disk.md0p1.in‚ 7.12109375 disk.md0.out‚ 444.75390625" proc.memory.rssMB‚ 3030.4765625…8H‚(:… î·çÔâªþ† gpu.0.gpu‚46.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚144.212' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.053 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@eµÂxž‚(:í î·çÔò¬Å‡ disk.sdb1.out‚0# disk./.usageGB‚2973.90629196167* proc.memory.availableMB‚10995.88671875) proc.memory.percent‚18.49656105041504 proc.cpu.threads‚17 network.sent‚ 8.097819e+06 disk.md0.out‚ 446.11328125 disk.sdb1.in‚0 disk.dm-0.out‚54755.42578125 disk.sda2.in‚3.625 cpu‚25.133047757483144 disk.rbd2.in‚0 disk.sdb.in‚ 3.49609375 disk.sda.out‚462.39794921875 disk.md0p1.out‚ 446.11328125" proc.memory.rssMB‚ 3030.4765625 disk.sdb2.in‚ 3.49609375 disk.rbd0.in‚0.015625 disk.rbd2.out‚1.0625 network.recv‚ 1.443704e+06 disk.md0.in‚ 7.12109375" disk.nvme1n1.in‚200391.3984375 disk.sda.in‚3.625# disk.nvme0n1.in‚328304.30078125 disk.dm-0.in‚528695.69921875! disk.sda2.out‚462.39794921875 disk.md0p1.in‚ 7.12109375% memory_percent‚32.886433601379395) disk./.usagePercent‚83.15166459972777! disk.sdb2.out‚462.39794921875# disk.nvme1n1.out‚36768.08203125 disk.nvme0n1.out‚ 17987.34375 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚ 0.837890625 disk.sdb.out‚462.39794921875Ì»¿u‘Ÿ‚(:† ý·çÔ¨¯ÿ… gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i”Ãðs ‚(:è ý·çÔ¼àþ†% disk./.usageGB‚2974.1568031311035 disk.rbd2.in‚0 disk.sda2.in‚3.625! disk.sdb2.out‚463.81689453125 disk.md0.in‚ 7.12109375 disk.rbd0.in‚0.015625 disk.sda1.out‚0) proc.memory.availableMB‚ 10996.1328125 proc.cpu.threads‚17 disk.dm-0.in‚ 528697.4375 disk.dm-0.out‚54871.25% memory_percent‚32.884931564331055 cpu‚25.14837449051538 disk.rbd0.out‚ 0.916015625 disk.sda.out‚463.81689453125 disk.sdb2.in‚ 3.49609375" proc.memory.rssMB‚ 3030.4765625 network.sent‚ 8.103783e+06 network.recv‚ 1.448087e+06 disk.sdb1.in‚0 disk.sda1.in‚0 disk.sdb.out‚463.81689453125 disk.sdb1.out‚0# disk.nvme0n1.out‚17987.95703125( disk./.usagePercent‚83.1586689968722# disk.nvme1n1.in‚200392.10546875 disk.sda.in‚3.625! disk.sda2.out‚463.81689453125 disk.md0p1.out‚ 447.4765625 disk.md0.out‚ 447.4765625 disk.rbd2.out‚ 1.07421875# disk.nvme1n1.out‚36883.29296875 disk.sdb.in‚ 3.49609375# disk.nvme0n1.in‚328305.33203125 disk.md0p1.in‚ 7.12109375) proc.memory.percent‚18.49656105041504ç q–2¡- _wandbruntime‚4278  _runtime‚4278]v®B*¢¼T FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚273 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚69006039 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 7208664193536 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 565025280 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 167663369 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚36 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20710 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚41242726 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188859828÷ÔUœ£ÁT `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 7208664193536 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚41242724 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚156559849675264 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 560288773 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 7134511104 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20712 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚163892833606144 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 151344128 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚273 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚98657328 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1300 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3532 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚147834719029451 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 171820468 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚98636618 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32005 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚163892658321920 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 572522496 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚309 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3532 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚36 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20710 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚41242726 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188859828 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚273 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚69006039 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 7208664193536 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 565025280 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 167663369 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32248 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚98645526 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚140672224011344 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚140672241050704 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 560288773 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚147839052682240 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6966900224 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3532 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚98666236 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚393 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚273 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20710 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32005 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 7209138053632 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚67469776 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 151344128 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚69021423 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚147839034679296 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚273 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1300 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚155048190735872 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1319 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 175284224 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚156590211428864 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚147860939746723 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188859828 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 7209138053632 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚26215761 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚41254408 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚582 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884682266001403e+09 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 167666947 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 565025280 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚147839052682240 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚67470169 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 7209138053632 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚67469776 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 151344128 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚69021423 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚147839034679296 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚273 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1300 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚155048190735872 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1319 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 175284224 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚156590211428864 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚147860939746723 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188859828 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 7209138053632 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚26215761 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚41254408 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚582 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884682266001403e+09 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 167666947 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 565025280 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚147839052682240 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚67470169 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32248 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚98645526 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚140672224011344 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚140672241050704 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 560288773 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚147839052682240 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6966900224 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3532 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0§£üˆ‘§‚(:† Œ¸çÔÍáó… gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ú)™ y¨‚(:î Œ¸çÔ…ŠÍˆ! disk.sda2.out‚465.31689453125 disk.sda.out‚465.31689453125! disk.nvme0n1.out‚ 17988.015625* proc.memory.availableMB‚10995.59765625 disk.md0p1.out‚ 448.91015625 disk.md0.in‚ 7.12109375 disk.sdb.in‚ 3.49609375* proc.memory.percent‚18.499183654785156 disk.sdb1.in‚0 network.sent‚ 8.109726e+06$ disk./.usageGB‚2974.407272338867 disk.sda1.in‚0! disk.sdb2.out‚465.31689453125 disk.rbd0.out‚ 0.916015625! disk.nvme1n1.in‚ 200392.359375% memory_percent‚32.888197898864746 cpu‚25.16316389265326 disk.sda1.out‚0 disk.sda2.in‚3.625 disk.dm-0.out‚ 55074.5390625 proc.memory.rssMB‚ 3030.90625 network.recv‚ 1.452371e+06) disk./.usagePercent‚83.16567222075079 disk.sdb2.in‚ 3.49609375 disk.md0.out‚ 448.91015625 disk.sdb.out‚465.31689453125 disk.dm-0.in‚528697.69140625 proc.cpu.threads‚17 disk.rbd2.in‚0" disk.nvme1n1.out‚ 37086.5234375 disk.sda.in‚3.625 disk.md0p1.in‚ 7.12109375 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 1.13671875 disk.sdb1.out‚0# disk.nvme0n1.in‚328305.332031254Z 2©- _wandbruntime‚4294  _runtime‚4294ýêK±ªæ¥ 7"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.52499920129776 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.27170222997665405 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3119428753852844 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18435494601726532 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2628782391548157 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9886180758476257 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 +jet/accuracy_3_of_3‚0.02253105491399765 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21734113991260529 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.22126440703868866 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5183201432228088 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18161563575267792 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5246692895889282 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4999999701976776 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.23046979308128357 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.33898472785949707 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0075813475996255875 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22000925242900848 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3194444477558136 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.2500306963920593 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 particle/accuracy_4_of_7‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 jet/accuracy_5_of_5‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4933333098888397 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.17565704882144928 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.02631578966975212 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08085108548402786 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.029587265104055405 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.24166667461395264 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3020833134651184 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.18210311233997345 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5052192211151123 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5767198801040649 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 particle/accuracy_2_of_6‚1 _runtime‚4294.390622323 8"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.529964029788971 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 particle/accuracy_3_of_4‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9975159764289856 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5789474248886108 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9979310631752014 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.037171605974435806 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6026122570037842 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.14133282005786896 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 jet/accuracy_5_of_7‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_4_of_5‚0.00770609313622117 particle/accuracy_2_of_7‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2566757798194885 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.3034650981426239 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.Òx ¬Ýx0987723246216774 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.26293104887008667 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15206767618656158 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.37104862928390503 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5130205750465393 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 particle/accuracy_1_of_4‚1 jet/accuracy_2_of_7‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11404433846473694 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09928574413061142 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ;"Purity/*FRt*SRqqt3FBt/event_purity‚0.021276595070958138 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 +jet/accuracy_3_of_5‚0.05480670928955078 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5753471851348877 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.021888386458158493 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05714285746216774 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9126984477043152 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.45782017707824707 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5518878102302551 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5157572627067566 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994527101516724 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5622038841247559 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2990495562553406 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 +jet/accuracy_3_of_6‚0.09615384787321091 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2793252468109131 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.12474862486124039 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_5‚1 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 8"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.589506208896637 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.43008068203926086 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5466195344924927 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8903509378433228 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.40612876415252686 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0821266919374466 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7677516341209412 particle/accuracy_1_of_5‚1 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5598715543746948 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 jet/accuracy_3_of_7‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5317502021789551 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.2002355456352234 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7575758099555969 particle/accuracy_1_of_6‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5509921312332153 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt3FBt/event_purity‚0.02380952425301075 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.1159454807639122 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18161563575267792 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15433663129806519 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 -jet/accuracy_4_of_4‚0.0034984340891242027 particle/accuracy_7_of_7‚0.5 _step‚296 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9912280440330505 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1160619705915451 $ _timestamp‚1.788468241203432e+09 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21861734986305237 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9860897660255432 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5103512406349182 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4217357635498047 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14476065337657928 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1277894228696823 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.4249999523162842 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9399999976158142 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4253411591053009 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2321428507566452 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5423114895820618 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0011337868636474013 (particle/accuracy‚0.8535206317901611 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5259374976158142 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5309568643569946 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4463904798030853 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 -"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3125 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1850244253873825 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9939104318618774 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9010416865348816 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996220469474792 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.42511096596717834 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_4_of_6‚0 jet/accuracy_4_of_7‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9905498623847961 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.49333861470222473 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3783781826496124 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15468186140060425 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.17798356711864471 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19893698394298553 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.025681523606181145 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5718390941619873 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4608756899833679 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8133881092071533 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.48903509974479675 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 particle/accuracy_1_of_7‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5423387289047241 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9808515906333923 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9628968834877014 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4087797999382019 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.3809523582458496 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9968628287315369 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1623452603816986 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3781144320964813 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_6_of_6‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17811724543571472 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9704861044883728 *jet/accuracy_1_of_4‚0.9212583303451538 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4929410219192505 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.43692201375961304 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 particle/accuracy_1_of_3‚1 *jet/accuracy_2_of_6‚0.5128205418586731 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2768060863018036 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9852729439735413 particle/accuracy_4_of_4‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4693806767463684 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9909665584564209 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 jet/accuracy_6_of_7‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.21474109590053558 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03222125023603439 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5944060683250427 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151411294937 jet/accuracy_7_of_7‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07396558672189713 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5759925246238708 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 +jet/accuracy_2_of_2‚0.10363775491714478 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4861111342906952 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1111111119389534 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.565014660358429 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7727272510528564 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1891675442457199 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9813876748085022 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02016703225672245 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.021094733849167824 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_6‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.03207547217607498 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900095462799072 +jet/accuracy_3_of_4‚0.07369408756494522 jet/accuracy_1_of_6‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.48336583375930786 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.18106059730052948 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.20987655222415924 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.992032527923584 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 particle/accuracy_5_of_5‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.4166666567325592 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 5 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.98918616771698 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0015873016091063619 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5381922125816345 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.26544979214668274 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2708333432674408 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5329148769378662 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 particle/accuracy_2_of_3‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14769576489925385 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21220819652080536 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4855895936489105 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06383348256349564 +jet/accuracy_2_of_4‚0.38462528586387634 'particle/f_score‚0.4118293225765228 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.39280369877815247 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9977278113365173 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10349920392036438 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.16695402562618256 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9901540875434875 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5165534019470215 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 epoch‚2 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9895125031471252 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.013888888992369175 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735632538795471 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_5‚0.9496095776557922 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.29012343287467957 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.02083333395421505 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5242100954055786 *jet/accuracy_1_of_3‚0.8822712302207947 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.369095116853714 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.009523809887468815 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.574404776096344 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.03125 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4274510145187378 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9702733755111694 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5426971912384033 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21787254512310028 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.03488372266292572 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 particle/accuracy_6_of_7‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.020378904417157173 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994527101516724 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.16111111640930176 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.05263157933950424 8"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.414923757314682 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5671703219413757 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 jet/accuracy_1_of_7‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4999999701976776 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6258314847946167 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.37627628445625305 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4273947477340698 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5896952152252197 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 )loss/val_total_loss‚44.01210406125627 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5456230640411377 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.37633925676345825 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2155405431985855 +particle/sensitivity‚0.3286844491958618 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.13555555045604706 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5659205317497253 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9905832409858704 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4389534890651703 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9899762868881226 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16372472047805786 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4972701370716095 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.012329554185271263 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5846026539802551 * Purity/2FRt1SRqqt2FBt/FRt_purity‚0.375 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4056040942668915 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19779913127422333 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2708333432674408 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.33026283979415894 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1501946747303009 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5837962627410889 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5166666507720947 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.014276765286922455 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.58134925365448 particle/accuracy_4_of_5‚1 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2750000059604645 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9899379014968872 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.543182373046875 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.28697800636291504 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 particle/accuracy_2_of_2‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5023148059844971 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1111111119389534 particle/accuracy_3_of_7‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.911347508430481 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4306458830833435 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 particle/accuracy_2_of_4‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 trainer/global_step‚3626 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13494905829429626 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5456230640411377 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.38148143887519836 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.42157742381095886 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5107803344726562 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9904787540435791 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4881603717803955 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12573866546154022 particle/accuracy_5_of_7‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.41620784997940063 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5445271730422974 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_1‚0.5050138831138611 +particle/specificity‚0.9506286382675171 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6980724930763245 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5ZyÏ «ë¥ :"Purity/*FRt0SRqqt*FBt/event_purity‚0.39280369877815247 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9977278113365173 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10349920392036438 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.16695402562618256 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9901540875434875 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5165534019470215 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 epoch‚2 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9895125031471252 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.013888888992369175 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735632538795471 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_5‚0.9496095776557922 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.29012343287467957 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.02083333395421505 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5242100954055786 *jet/accuracy_1_of_3‚0.8822712302207947 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.369095116853714 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.009523809887468815 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.574404776096344 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.03125 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4274510145187378 ?&Puöéåùrity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9702733755111694 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5426971912384033 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21787254512310028 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.03488372266292572 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 particle/accuracy_6_of_7‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.020378904417157173 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994527101516724 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.16111111640930176 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.05263157933950424 8"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.414923757314682 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5671703219413757 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 jet/accuracy_1_of_7‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4999999701976776 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6258314847946167 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.37627628445625305 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4273947477340698 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5896952152252197 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 )loss/val_total_loss‚44.01210406125627 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5456230640411377 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.37633925676345825 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2155405431985855 +particle/sensitivity‚0.3286844491958618 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.13555555045604706 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5659205317497253 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9905832409858704 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4389534890651703 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9899762868881226 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16372472047805786 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4972701370716095 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.012329554185271263 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5846026539802551 * Purity/2FRt1SRqqt2FBt/FRt_purity‚0.375 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4056040942668915 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19779913127422333 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2708333432674408 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.33026283979415894 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1501946747303009 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5837962627410889 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5166666507720947 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.014276765286922455 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.58134925365448 particle/accuracy_4_of_5‚1 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2750000059604645 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9899379014968872 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.543182373046875 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.28697800636291504 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 particle/accuracy_2_of_2‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5023148059844971 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1111111119389534 particle/accuracy_3_of_7‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.911347508430481 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4306458830833435 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 particle/accuracy_2_of_4‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 trainer/global_step‚3626 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13494905829429626 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5456230640411377 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.38148143887519836 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.42157742381095886 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5107803344726562 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9904787540435791 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4881603717803955 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12573866546154022 particle/accuracy_5_of_7‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.41620784997940063 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5445271730422974 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_1‚0.5050138831138611 +particle/specificity‚0.9506286382675171 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6980724930763245 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 7"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.52499920129776 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.27170222997665405 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3119428753852844 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18435494601726532 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2628782391548157 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9886180758476257 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 +jet/accuracy_3_of_3‚0.02253105491399765 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21734113991260529 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.22126440703868866 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5183201432228088 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18161563575267792 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5246692895889282 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4999999701976776 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.23046979308128357 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.33898472785949707 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0075813475996255875 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22000925242900848 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3194444477558136 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.2500306963920593 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 particle/accuracy_4_of_7‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 jet/accuracy_5_of_5‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4933333098888397 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.17565704882144928 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.02631578966975212 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08085108548402786 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.029587265104055405 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.24166667461395264 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3020833134651184 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.18210311233997345 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5052192211151123 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5767198801040649 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 particle/accuracy_2_of_6‚1 _runtime‚4294.390622323 8"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.529964029788971 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 particle/accuracy_3_of_4‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9975159764289856 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5789474248886108 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9979310631752014 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.037171605974435806 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6026122570037842 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.14133282005786896 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 jet/accuracy_5_of_7‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_4_of_5‚0.00770609313622117 particle/accuracy_2_of_7‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2566757798194885 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.3034650981426239 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.26293104887008667 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15206767618656158 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.37104862928390503 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5130205750465393 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 particle/accuracy_1_of_4‚1 jet/accuracy_2_of_7‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11404433846473694 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09928574413061142 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ;"Purity/*FRt*SRqqt3FBt/event_purity‚0.021276595070958138 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 +jet/accuracy_3_of_5‚0.05480670928955078 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5753471851348877 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.021888386458158493 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05714285746216774 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9126984477043152 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.45782017707824707 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5518878102302551 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5157572627067566 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994527101516724 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5622038841247559 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2990495562553406 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 +jet/accuracy_3_of_6‚0.09615384787321091 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2793252468109131 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.12474862486124039 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_5‚1 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 8"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.589506208896637 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.43008068203926086 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5466195344924927 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8903509378433228 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.40612876415252686 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0821266919374466 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7677516341209412 particle/accuracy_1_of_5‚1 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5598715543746948 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 jet/accuracy_3_of_7‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5317502021789551 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.2002355456352234 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7575758099555969 particle/accuracy_1_of_6‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5509921312332153 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt3FBt/event_purity‚0.02380952425301075 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.1159454807639122 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18161563575267792 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15433663129806519 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 -jet/accuracy_4_of_4‚0.0034984340891242027 particle/accuracy_7_of_7‚0.5 _step‚296 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9912280440330505 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1160619705915451 $ _timestamp‚1.788468241203432e+09 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21861734986305237 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9860897660255432 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5103512406349182 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4217357635498047 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14476065337657928 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1277894228696823 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.4249999523162842 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9399999976158142 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4253411591053009 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2321428507566452 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5423114895820618 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0011337868636474013 (particle/accuracy‚0.8535206317901611 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5259374976158142 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5309568643569946 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4463904798030853 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 -"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3125 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1850244253873825 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9939104318618774 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9010416865348816 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996220469474792 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.42511096596717834 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_4_of_6‚0 jet/accuracy_4_of_7‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9905498623847961 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.49333861470222473 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3783781826496124 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15468186140060425 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.17798356711864471 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19893698394298553 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.025681523606181145 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5718390941619873 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4608756899833679 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8133881092071533 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.48903509974479675 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 particle/accuracy_1_of_7‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5423387289047241 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9808515906333923 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9628968834877014 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4087797999382019 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.3809523582458496 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9968628287315369 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1623452603816986 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3781144320964813 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_6_of_6‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17811724543571472 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9704861044883728 *jet/accuracy_1_of_4‚0.9212583303451538 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4929410219192505 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.43692201375961304 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 particle/accuracy_1_of_3‚1 *jet/accuracy_2_of_6‚0.5128205418586731 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2768060863018036 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9852729439735413 particle/accuracy_4_of_4‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4693806767463684 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9909665584564209 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 jet/accuracy_6_of_7‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.21474109590053558 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03222125023603439 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5944060683250427 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151411294937 jet/accuracy_7_of_7‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07396558672189713 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5759925246238708 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 +jet/accuracy_2_of_2‚0.10363775491714478 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4861111342906952 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1111111119389534 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.565014660358429 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7727272510528564 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1891675442457199 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9813876748085022 @&Purity/0FRt3SRqqF1õÓä t1FBt/event_proportion‚0.0006200397037900984 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02016703225672245 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.021094733849167824 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_6‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.03207547217607498 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900095462799072 +jet/accuracy_3_of_4‚0.07369408756494522 jet/accuracy_1_of_6‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.48336583375930786 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.18106059730052948 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.20987655222415924 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.992032527923584 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 particle/accuracy_5_of_5‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.4166666567325592 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 5 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.98918616771698 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0015873016091063619 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5381922125816345 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.26544979214668274 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2708333432674408 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5329148769378662 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 particle/accuracy_2_of_3‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14769576489925385 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21220819652080536 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4855895936489105 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06383348256349564 +jet/accuracy_2_of_4‚0.38462528586387634 'particle/f_score‚0.4118293225765228 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1¨y󒬂(:… ›¸çÔ諾† gpu.0.gpu‚12.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚144.212' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.053 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æ6 ?u­‚(:ê ›¸çÔÑé† disk.rbd2.in‚0 disk.sdb.in‚ 3.49609375 disk.sdb.out‚467.26806640625 proc.memory.rssMB‚3030.25 disk.rbd0.in‚0.015625" disk.nvme0n1.out‚ 17988.1484375$ memory_percent‚33.92651081085205 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.availableMB‚10825.48046875) proc.memory.percent‚18.49517822265625 cpu‚25.166943142602907! disk.sdb2.out‚467.26806640625 disk.dm-0.in‚528697.69140625 disk.sda2.in‚3.625 proc.cpu.threads‚17) disk./.usagePercent‚83.16263975516203 disk.sdb2.in‚ 3.49609375 disk.md0.in‚ 7.12109375! disk.nvme1n1.in‚ 200392.359375 disk.md0.out‚ 450.7890625 disk.rbd0.out‚ 1.01806640625 disk.sda.out‚467.26806640625 disk.dm-0.out‚ 55631.828125 disk.md0p1.out‚ 450.7890625" disk.nvme1n1.out‚ 37643.6796875 disk.sdb1.in‚0 disk.sdb1.out‚0# disk.nvme0n1.in‚328305.33203125 disk.rbd2.out‚ 1.13671875 disk.sda.in‚3.625! disk.sda2.out‚467.26806640625 disk.md0p1.in‚ 7.12109375 network.sent‚ 8.216226e+06 network.recv‚ 1.458446e+06$ disk./.usageGB‚2974.298816680908£2Wf®‚(:Û ª¸çÔü‡ò† disk.sdb2.out‚ 468.7578125 disk.rbd0.in‚0.015625# disk.nvme1n1.out‚37810.59765625 network.recv‚ 1.465603e+06 disk.md0.out‚ 452.234375 disk.rbd0.out‚ 1.01806640625 disk.rbd2.out‚ 1.2158203125 disk.sdb.in‚ 3.49609375# disk.nvme0n1.in‚328305.33203125 disk.dm-0.in‚528697.69140625* proc.memory.availableMB‚10825.48828125 cpu‚25.16642452454341 disk.md0p1.out‚ 452.234375) proc.memory.percent‚18.49517822265625 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sdb2.in‚ 3.49609375 disk.sdb.out‚ 468.7578125 network.sent‚ 8.339519e+06! disk.nvme1n1.in‚ 200392.359375 disk.sdb1.out‚0! disk.nvme0n1.out‚ 17988.203125 disk.md0.in‚ 7.12109375 disk.sda1.out‚0 disk.sda2.in‚3.625 disk.sda2.out‚ 468.7578125 disk.rbd2.in‚0$ memory_percent‚33.92646312713623 proc.memory.rssMB‚3030.25 disk.md0p1.in‚ 7.12109375 disk.dm-0.out‚55798.80078125 disk.sdb1.in‚0 disk.sda.in‚3.625 disk.sda.out‚ 468.7578125( disk./.usagePercent‚83.1766123915305% disk./.usageGB‚2974.7985458374023Äy§Ð¯‚(:… ª¸çԽݫˆ gpu.0.gpu‚10.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚144.212' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.053 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NÒHæ2°- _wandbruntime‚4321  _runtime‚43210ÖƱÀ _step‚297 & lr-AdamW/pg1‚0.0007263206089044752 & lr-AdamW/pg2‚0.0007263206089044752 trainer/global_step‚3649 $ _timestamp‚1.788468268619383e+09 _runtime‚4321.804412718@/S˲Š& lr-AdamW/pg2‚0.0007263206089044752 trainer/global_step‚3649 $ _timestamp‚1.788468268619383e+09 _runtime‚4321.804412718 _step‚297 & lr-AdamW/pg1‚0.0007263206089044752©:¥Ñ¬2³- _wandbruntime‚4321  _runtime‚4321ókþõÜ'´ÖO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41643916 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 299008000 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7258284742656 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7383357088256 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 168997518 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚157901985506816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚298 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚664 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚165285318654976 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26448914 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 _runtime‚4321.805778144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚149094199261184 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 trainer/global_step‚3649 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156352466000896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7237933950656 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚149094181258240 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7258284742656 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41643918 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚149094199261184 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚156352466000896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69623437 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 % _timestamp‚1.7884682686205668e+09 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚149094181258240JßYÚá'µÛO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156352466000896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7237933950656 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚149094181258240 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7258284742656 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41643918 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚149094199261184 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚156352466000896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69623437 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 % _timestamp‚1.7884682686205668e+09 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚149094181258240 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41643916 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 299008000 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7258284742656 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7383357088256 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 168997518 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚157901985506816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚298 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚664 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚165285318654976 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26448914 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 _runtime‚4321.805778144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚149094199261184 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 trainer/global_step‚3649ªÖ ÚE2¶- _wandbruntime‚4322  _runtime‚4322}A*â·ÞM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚156393967748608 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚149107607156522 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚141867714716672 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69641712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169042511 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26455870 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚141867697677312 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7239663458558 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169016139 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68110588 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚157943645536768 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚149107361135870 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7260248768512 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚157943669476864 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚165329308553216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7260482450432 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169016139 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7385639076352 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68111264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69641714 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41654718 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169042511 Pq.all.freed‚156393716063744 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7260482450432 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26456544 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7385354761728 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99374427 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99374427 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 # _timestamp‚1.78846826969128e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚149133467295232 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚149133467295232 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7260248768512 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7239892439850 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚149133485298176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41654720 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚149133485298176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3649 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69641712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 5725224969©c(é&¸ãM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚156393967748608 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚149107607156522 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚141867714716672 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69641712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169042511 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26455870 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚141867697677312 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7239663458558 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169016139 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68110588 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚157943645536768 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚149107361135870 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7260248768512 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚157943669476864 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚165329308553216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7260482450432 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169016139 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7385639076352 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68111264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69641714 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41654718 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169042511 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99374427 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99374427 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 # _timestamp‚1.78846826969128e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚149133467295232 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚149133467295232 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7260248768512 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7239892439850 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚149133485298176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41654720 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚149133485298176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3649 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69641712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496«3ŒWþ2¹- _wandbruntime‚4322  _runtime‚4322ÙiBZºÔ 2loss/SRqqt3/assignment_loss‚0.9075774141238071 /loss/FRt3/detection_loss‚1.6073457582454596 0loss/SRqqt4/detection_loss‚2.598843951225281 /loss/FRt4/assignment_loss‚4.270007448811685 2loss/SRqqt4/assignment_loss‚1.7125264006853103 1loss/SRqqt1/detection_loss‚1.8910733989949495 trainer/global_step‚3649 1loss/SRqqt1/assignment_loss‚1.771816110695308 1loss/FBt2/assignment_loss‚0.10948766631814319 2loss/FBt4/assignment_loss‚0.048773372068270954 /loss/FBt4/detection_loss‚2.9044398936358364 _step‚300 0loss/FRt2/assignment_loss‚3.5525928304550494 epoch‚3 0loss/FRt3/assignment_loss‚1.6475562274456024 0loss/SRqqt2/detection_loss‚2.671910828777722 /loss/FBt2/detection_loss‚2.7483094313314984 /loss/FBt3/detection_loss‚0.6023990425351515 0loss/FBt1/assignment_loss‚0.5024584887665924 .loss/FRt1/detection_loss‚2.633458021210461 .loss/FRt2/detection_loss‚2.103459641713887 .loss/FRt4/detection_loss‚3.066036638713652 1loss/SRqqt3/detection_loss‚0.9981374727018798 &loss/total_loss‚45.595219858404334 _runtime‚4322.878017761 1loss/FBt3/assignment_loss‚0.11936457904687506 .loss/FBt1/detection_loss‚2.272717082872987 $ _timestamp‚1.788468269693302e+09 /loss/FRt1/assignment_loss‚3.156949906814389 2loss/SRqqt2/assignment_loss‚1.6979782512145383ûª*$_»Ù /loss/FRt1/assignment_loss‚3.156949906814389 2loss/SRqqt2/assignment_loss‚1.6979782512145383 2loss/SRqqt3/assignment_loss‚0.9075774141238071 /loss/FRt3/detection_loss‚1.6073457582454596 0loss/SRqqt4/detection_loss‚2.598843951225281 /loss/FRt4/assignment_loss‚4.270007448811685 2loss/SRqqt4/assignment_loss‚1.7125264006853103 1loss/SRqqt1/detection_loss‚1.8910733989949495 trainer/global_step‚3649 1loss/SRqqt1/assignment_loss‚1.771816110695308 1loss/FBt2/assignment_loss‚0.10948766631814319 2loss/FBt4/assignment_loss‚0.048773372068270954 /loss/FBt4/detection_loss‚2.9044398936358364 _step‚300 0loss/FRt2/assignment_loss‚3.5525928304550494 epoch‚3 0loss/FRt3/assignment_loss‚1.6475562274456024 0loss/SRqqt2/detection_loss‚2.671910828777722 /loss/FBt2/detection_loss‚2.7483094313314984 /loss/FBt3/detection_loss‚0.6023990425351515 0loss/FBt1/assignment_loss‚0.5024584887665924 .loss/FRt1/detection_loss‚2.633458021210461 .loss/FRt2/detection_loss‚2.103459641713887 .loss/FRt4/detection_loss‚3.066036638713652 1loss/SRqqt3/detection_loss‚0.9981374727018798 &loss/total_loss‚45.595219858404334 _runtime‚4322.878017761 1loss/FBt3/assignment_loss‚0.11936457904687506 .loss/FBt1/detection_loss‚2.272717082872987 $ _timestamp‚1.788468269693302e+09¬ c’¼‚(:‡ ¹¸çÔÕï‡ gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚134.575' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.64375 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆåÓær½‚(:ç ¹¸çÔÛ½·ˆ disk.rbd2.out‚ 1.2158203125 disk.sda.out‚470.63232421875 disk.sda1.out‚0! disk.sdb2.out‚470.63232421875 disk.sdb.out‚470.63232421875) proc.memory.percent‚18.49517822265625 proc.cpu.threads‚17) disk./.usagePercent‚83.17723838219354 disk.rbd0.out‚ 1.06591796875 disk.md0p1.out‚ 454.046875 proc.memory.rssMB‚3030.25 network.sent‚ 8.345327e+06% disk./.usageGB‚2974.8209342956543 disk.md0.in‚ 7.12109375 disk.md0.out‚ 454.046875 disk.rbd0.in‚0.015625 disk.dm-0.out‚ 56114.375 disk.sda1.in‚0 disk.md0p1.in‚ 7.12109375! disk.nvme1n1.in‚ 200392.359375" disk.nvme0n1.out‚ 17988.4921875 disk.sda2.in‚3.625 disk.sdb2.in‚ 3.49609375" disk.nvme1n1.out‚ 38125.8828125 disk.sdb1.in‚0 disk.dm-0.in‚528697.69140625! disk.sda2.out‚470.63232421875 network.recv‚ 1.469645e+06 disk.sdb1.out‚0 disk.sda.in‚3.625# disk.nvme0n1.in‚328305.33203125 cpu‚25.165948960428093 disk.rbd2.in‚0 disk.sdb.in‚ 3.49609375$ memory_percent‚33.92643928527832) proc.memory.availableMB‚ 10825.49218751Ç&7’¾‚(:‡ ȸçÔ÷à’ˆ gpu.0.gpu‚10.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚142.313' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.57825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„zóñ}¿‚(:ò ȸçÔۛ܈ disk.sda2.in‚3.625 disk.rbd0.in‚0.015625! disk.nvme1n1.in‚ 200392.359375 disk.sdb.in‚ 3.49609375 disk.sda.out‚472.11181640625$ memory_percent‚33.92622470855713! disk.sda2.out‚472.11181640625 disk.md0p1.out‚ 455.48046875 disk.sda.in‚3.625 disk.dm-0.in‚528697.69140625 disk.dm-0.out‚56275.08203125* proc.memory.availableMB‚10825.52734375 proc.memory.rssMB‚3030.25) disk./.usagePercent‚83.17904467830502 disk.sda1.in‚0 disk.rbd0.out‚ 1.06591796875" disk.nvme1n1.out‚ 38286.5234375 disk.sdb1.out‚0# disk.nvme0n1.in‚328305.33203125 disk.md0.in‚ 7.12109375 disk.rbd2.in‚0# disk.nvme0n1.out‚17988.55859375 network.recv‚ 1.476045e+06% disk./.usageGB‚2974.8855361938477 cpu‚25.16543455982004 disk.sdb2.in‚ 3.49609375 network.sent‚ 8.452641e+06 disk.sda1.out‚0 disk.md0p1.in‚ 7.12109375! disk.sdb2.out‚472.11181640625 disk.rbd2.out‚ 1.27880859375 disk.sdb.out‚472.11181640625 disk.sdb1.in‚0) proc.memory.percent‚18.49517822265625 disk.md0.out‚ 455.48046875 proc.cpu.threads‚17é ~Ò‘À‚(:† ׸çÔχՇ gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚118.142' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.5355 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)*ÔxÁ‚(:í ׸çÔ€Ó͈! disk.sda2.out‚473.62841796875 disk.md0.in‚ 7.12109375 disk.rbd2.in‚0" disk.nvme1n1.out‚ 38629.0078125 disk.sda.out‚473.62841796875 proc.cpu.threads‚17 disk.sdb2.in‚ 3.49609375 disk.sdb.in‚ 3.49609375% memory_percent‚33.928513526916504 proc.memory.rssMB‚3030.25 disk.sda1.out‚0! disk.nvme1n1.in‚ 200392.359375 network.sent‚ 8.46023e+06 disk.rbd0.in‚0.015625 disk.sdb1.out‚0 cpu‚25.16487197771674 disk.rbd0.out‚ 1.12890625 disk.sdb1.in‚0 disk.sda.in‚3.625 disk.md0p1.in‚ 7.12109375 disk.md0.out‚ 456.94140625) proc.memory.percent‚18.49517822265625' disk./.usagePercent‚83.184297709512 disk.sda1.in‚0 disk.md0p1.out‚ 456.94140625 disk.sdb.out‚473.62841796875# disk.nvme0n1.out‚17988.74609375 network.recv‚ 1.479634e+06 disk.dm-0.out‚56617.75390625 disk.sda2.in‚3.625! disk.sdb2.out‚473.62841796875 disk.rbd2.out‚ 1.27880859375# disk.nvme0n1.in‚328305.33203125* proc.memory.availableMB‚10825.15234375% disk./.usageGB‚2975.0734100341797 disk.dm-0.in‚528697.69140625èµ`/2Â- _wandbruntime‚4378  _runtime‚43789 'ÇÃÁ _step‚301 & lr-AdamW/pg1‚0.0007224339338812805 & lr-AdamW/pg2‚0.0007224339338812805 trainer/global_step‚3699 % _timestamp‚1.7884683251986294e+09 _runtime‚4378.383569127¥H”ÌÄÆ _runtime‚4378.383569127 _step‚301 & lr-AdamW/pg1‚0.0007224339338812805 & lr-AdamW/pg2‚0.0007224339338812805 trainer/global_step‚3699 % _timestamp‚1.7884683251986294e+09­t 52Å- _wandbruntime‚4378  _runtime‚4378#É=üà'ÆÚO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7335871991164 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100712031 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70537187 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚158415235228672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100712031 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚158414983557120 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚699 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70537189 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7356500449280 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚151042574221612 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 % _timestamp‚1.7884683251994174e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 171249220 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 trainer/global_step‚3699 _step‚302 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68981293 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7483608890368 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚167469478340096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚151042328201168 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚158415235228672 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚143706702230448 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100685687 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚158414983557120 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68981994 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42184016 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚159986184009216 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7483318270976 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100685687 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7356734117888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42184018 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚151058501110784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚151058501110784 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚151058483107840 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70537189 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26797976µò¤ÀÇßO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ]IDeviceStatsMonitor.on_train_baļÁâ%'tch_start/active_bytes.large_pool.allocated‚151058501110784 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚151058483107840 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70537189 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26797976 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7335871991164 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100712031 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70537187 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚158415235228672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100712031 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚158414983557120 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚699 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70537189 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7356500449280 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚151042574221612 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 % _timestamp‚1.7884683251994174e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 171249220 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 trainer/global_step‚3699 _step‚302 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68981293 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7483608890368 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚167469478340096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚151042328201168 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚158415235228672 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚143706702230448 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100685687 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚158414983557120 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68981994 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42184016 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚159986184009216 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7483318270976 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100685687 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7356734117888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42184018 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚151058501110784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70555462 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100738749 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70555462 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171294213 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7337601499066 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 171294213 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚693 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚167513470087680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26804594 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚151097769144832 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚695 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚143744232587872 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100712379 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚160027867979264 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171267841 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7358698157056 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚158456485304832 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 171267841 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26805287 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚143744215548512 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚160027844039168 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7358464475136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚151082063068230 trainer/global_step‚3699 _runtime‚4379.441992236 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100738749 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69000107 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚167513149250048 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚158456233619968 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7358698157056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚151081817047578 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68999412 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚151097769144832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70555462 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100738749 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70555462 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171294213 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7337601499066 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 171294213 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚693 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚167513470087680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26804594 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚151097769144832 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚695 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚143744232587872 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100712379 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚160027867979264 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171267841 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7358698157056 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚158456485304832 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 171267841 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26805287 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚143744215548512 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚160027844039168 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7358464475136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚151082063068230 trainer/global_step‚3699 _runtime‚4379.441992236 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100738749 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69000107 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚167513149250048 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚158456233619968 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7358698157056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚151081817047578 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68999412 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚151097769144832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _step‚306 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7454716155904 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7433810031672 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7433581050588 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚153022784957440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚169653835820032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102023639 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚153022802960384 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7454949824512 trainer/global_step‚3749 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚162070382511616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102049983 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71450939 $ _timestamp‚1.788468379879148e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160477501113344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚153022784957440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173500922 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚145583203062288 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚688 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42724118 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27149342 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 296910848 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚160477501113344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42724116 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚162070358571520 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚160477752784896 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _step‚306 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7454716155904 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7433810031672 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7433581050588 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚153022784957440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚169653835820032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102023639 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚153022802960384 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7454949824512 trainer/global_step‚3749 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚162070382511616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102049983 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71450939 $ _timestamp‚1.788468379879148e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160477501113344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚153022784957440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173500922 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚145583203062288 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚688 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42724118 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27149342 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 296910848 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚160477501113344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42724116 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚162070358571520 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚160477752784896 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71469214 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚162112066481664 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7456680181760 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚160518751176192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27157094 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102076701 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚153062088997376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173545915 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚169697510281216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚153062070994432 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27156423 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 trainer/global_step‚3749 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocah¢º>€ted‚160519002861056 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69892014 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7435768520866 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚671 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102050331 _step‚307 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7456913863680 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71469212 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42734920 % _timestamp‚1.7884683809590378e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173519543 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7585443799552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚153056518979938 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚145620750459072 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7435539539574 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚169697189443584 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173519543 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69891341 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7456680181760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249† 3nï&ÞéM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚160519002861056 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69891341 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7456680181760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71469214 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚162112066481664 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7456680181760 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚160518751176192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27157094 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102076701 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚153062088997376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173545915 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚169697510281216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚153062070994432 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27156423 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 trainer/global_step‚3749 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚160519002861056 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69892014 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7435768520866 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚671 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102050331 _step‚307 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7456913863680 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71469212 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42734920 % _timestamp‚1.7884683809590378e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173519543 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7585443799552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚153056518979938 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚145620750459072 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7435539539574 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚169697189443584 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173519543 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚162540018669568 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚162540270341120 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72364689 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72364687 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7682949315072 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚154991240024584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚3799 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚154991486045028 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7553165531136 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚164154557073920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚171837820948480 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72364689 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27496334 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚164154581014016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103361591 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚154987104809984 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7553165531136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27497027 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚162540018669568 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚162540270341120 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72364689 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72364687 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7682949315072 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚154991240024584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚3799 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚154991486045028 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7553165531136 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚164154557073920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚171837820948480 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72364689 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27496334 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚164154581014016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103361591 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚154987104809984 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7553165531136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27497027 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚171881183707136 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7554895888384 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175797617 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚155026390846976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175797617 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103414653 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70778746 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚155026372844032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚155026372844032 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70778064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43275020 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7533706561374 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚162581520417280 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7555129570304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚162581520417280 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7685229074944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚147497268330272 % _timestamp‚1.7884684354948127e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚682 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 trainer/global_step‚3799 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72382964 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72382964 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚164196241043968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚171881183707136 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7554895888384 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175797617 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚155026390846976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175797617 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103414653 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70778746 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚155026372844032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚155026372844032 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70778064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43275020 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7533706561374 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚162581520417280 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7555129570304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚162581520417280 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7685229074944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚147497268330272 % _timestamp‚1.7884684354948127e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚682 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 trainer/global_step‚3799 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72382964·C’y2ó- _wandbruntime‚4488  _runtime‚4488Á|Ë1ìôÏ /loss/FRt4/detection_loss‚2.9786364610852867 2loss/SRqqt4/assignment_loss‚1.6616025124159124 1loss/FBt2/assignment_loss‚0.23588787651783605 .loss/FBt1/detection_loss‚2.698137104511261 $ _timestamp‚1.788468435496863e+09 _runtime‚4488.681620312 0loss/FBt4/assignment_loss‚0.6503987468030655 1loss/SRqqt3/detection_loss‚0.8190940156279282 trainer/global_step‚3799 0loss/FRt1/assignment_loss‚3.5690488636493685 /loss/FBt3/detection_loss‚0.7933382781511041 %loss/total_loss‚44.31262288482887 0loss/SRqqt1/assignment_loss‚1.97059521519325 2loss/SRqqt2/assignment_loss‚1.4802646282478236 /loss/FRt2/detection_loss‚2.0552143836114554 /loss/FBt2/detection_loss‚1.7837863993644714 2loss/SRqqt3/assignment_loss‚0.8662077442946277 1loss/FBt3/assignment_loss‚0.02416239487583848 0loss/SRqqt1/detection_loss‚1.845641939477487 /loss/FRt2/assignment_loss‚3.552825622558594 0loss/FBt1/assignment_loss‚0.6942700387632034 /loss/SÓÃSOiRqqt4/detection_loss‚2.86053615698108 .loss/FBt4/detection_loss‚2.680631334131414 epoch‚3 0loss/FRt3/assignment_loss‚1.2200250503135055 1loss/SRqqt2/detection_loss‚2.2674941271543503 _step‚312 .loss/FRt4/assignment_loss‚3.70302585486708 .loss/FRt1/detection_loss‚2.641633625825246 /loss/FRt3/detection_loss‚1.2601645104076764}·½öZõÔ 0loss/SRqqt1/assignment_loss‚1.97059521519325 2loss/SRqqt2/assignment_loss‚1.4802646282478236 /loss/FRt2/detection_loss‚2.0552143836114554 /loss/FBt2/detection_loss‚1.7837863993644714 2loss/SRqqt3/assignment_loss‚0.8662077442946277 1loss/FBt3/assignment_loss‚0.02416239487583848 0loss/SRqqt1/detection_loss‚1.845641939477487 /loss/FRt2/assignment_loss‚3.552825622558594 0loss/FBt1/assignment_loss‚0.6942700387632034 /loss/SRqqt4/detection_loss‚2.86053615698108 .loss/FBt4/detection_loss‚2.680631334131414 epoch‚3 0loss/FRt3/assignment_loss‚1.2200250503135055 1loss/SRqqt2/detection_loss‚2.2674941271543503 _step‚312 .loss/FRt4/assignment_loss‚3.70302585486708 .loss/FRt1/detection_loss‚2.641633625825246 /loss/FRt3/detection_loss‚1.2601645104076764 /loss/FRt4/detection_loss‚2.9786364610852867 2loss/SRqqt4/assignment_loss‚1.6616025124159124 1loss/FBt2/assignment_loss‚0.23588787651783605 .loss/FBt1/detection_loss‚2.698137104511261 $ _timestamp‚1.788468435496863e+09 _runtime‚4488.681620312 0loss/FBt4/assignment_loss‚0.6503987468030655 1loss/SRqqt3/detection_loss‚0.8190940156279282 trainer/global_step‚3799 0loss/FRt1/assignment_loss‚3.5690488636493685 /loss/FBt3/detection_loss‚0.7933382781511041 %loss/total_loss‚44.31262288482887¸»cmœö‚(:‘ Þ¹çÔž™´‰ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00n_v÷‚(:ë Þ¹çÔé艊 network.sent‚ 8.713374e+06 network.recv‚ 1.522185e+06 disk.md0p1.out‚ 471.05859375 disk.md0.in‚ 7.12109375 disk.rbd0.out‚ 1.19189453125$ memory_percent‚34.85095500946045* proc.memory.availableMB‚10674.01953125 disk.sda1.in‚0 disk.sda2.out‚488.2607421875 disk.sdb2.out‚488.2607421875 disk.rbd2.in‚0# disk.nvme1n1.out‚40153.90234375) proc.memory.percent‚18.49517822265625# disk.nvme0n1.out‚18363.80078125 disk.dm-0.out‚ 58517.703125 disk.rbd0.in‚0.015625 proc.memory.rssMB‚3030.25# disk.nvme0n1.in‚328305.47265625 disk.rbd2.out‚ 1.341796875 disk.sdb.in‚ 3.49609375 disk.sdb1.in‚0 disk.sda.out‚488.2607421875 disk.sdb1.out‚0 disk.sda.in‚3.625 disk.sda1.out‚0 disk.sda2.in‚3.625 disk.md0p1.in‚ 7.12109375 disk.md0.out‚ 471.05859375! disk.nvme1n1.in‚ 200393.109375 disk.sdb.out‚488.2607421875 cpu‚25.16077208005135 proc.cpu.threads‚17) disk./.usagePercent‚83.15466368056997% disk./.usageGB‚2974.0135536193848 disk.dm-0.in‚528698.58203125 disk.sdb2.in‚ 3.49609375ƈçï’ø‚(:‡ í¹çÔØç…‡ gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚144.821' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.20525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0r{¥szù‚(:ï í¹çÔô°î‡ disk.md0.out‚ 472.44921875 disk.sda.out‚489.71630859375 disk.md0p1.out‚ 472.44921875$ memory_percent‚34.97910499572754 cpu‚25.160322516278676 proc.cpu.threads‚17 network.sent‚ 8.823005e+06 network.recv‚ 1.529032e+06) disk./.usagePercent‚83.14256474976928! disk.nvme1n1.in‚ 200393.109375 disk.sdb.in‚ 3.49609375 disk.sda.in‚3.625 disk.sda1.out‚0# disk.nvme0n1.out‚18473.87890625 disk.sda2.in‚3.625 disk.rbd0.out‚ 1.19189453125 disk.sdb.out‚489.71630859375 disk.sdb1.in‚0 disk.sda1.in‚0 disk.sdb2.in‚ 3.49609375 disk.rbd0.in‚0.015625 disk.dm-0.out‚ 58634.4921875 disk.md0p1.in‚ 7.12109375) proc.memory.percent‚18.49517822265625 disk.dm-0.in‚528698.58203125$ disk./.usageGB‚2973.580837249756! disk.sdb2.out‚489.71630859375 disk.md0.in‚ 7.12109375 disk.rbd2.in‚0 disk.rbd2.out‚ 1.341796875# disk.nvme1n1.out‚40160.61328125# disk.nvme0n1.in‚328305.47265625 disk.sdb1.out‚0! disk.sda2.out‚489.71630859375) proc.memory.availableMB‚ 10653.0234375 proc.memory.rssMB‚3030.25ƒŸ=’ú‚(:‡ ü¹çÔÿ힇 gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚116.583' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.14575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jÀtû‚(:é ü¹çÔÝ¥Ö‡) proc.memory.percent‚18.49517822265625 disk.md0.in‚ 7.12109375 disk.sda.out‚491.45751953125) proc.memory.availableMB‚ 10653.7578125 cpu‚25.159852552622603 disk.dm-0.in‚528698.58203125 disk.sda2.in‚3.625 disk.rbd2.in‚0 disk.sdb1.out‚0 proc.cpu.threads‚17# disk.nvme0n1.in‚328305.47265625! disk.nvme0n1.out‚ 18602.703125 disk.sda1.in‚0 disk.sdb2.in‚ 3.49609375$ memory_percent‚34.97462272644043 proc.memory.rssMB‚3030.25 network.recv‚ 1.53342e+06 disk.sda1.out‚0! disk.sdb2.out‚491.45751953125 disk.rbd2.out‚ 1.341796875# disk.nvme1n1.out‚40160.72265625) disk./.usagePercent‚83.14608145417385$ disk./.usageGB‚2973.706611633301! disk.sda2.out‚491.45751953125 disk.sdb.out‚491.45751953125 network.sent‚ 8.826846e+06 disk.md0p1.out‚ 474.140625 disk.md0.out‚ 474.140625 disk.rbd0.in‚0.015625 disk.dm-0.out‚58763.42578125 disk.md0p1.in‚ 7.12109375 disk.rbd0.out‚ 1.19189453125! disk.nvme1n1.in‚ 200393.109375 disk.sdb.in‚ 3.49609375 disk.sdb1.in‚0 disk.sda.in‚3.625JýKÜ2ü- _wandbruntime‚4540  _runtime‚4540ó$ûÇýÁ & lr-AdamW/pg2‚0.0007103893351387208 trainer/global_step‚3849 % _timestamp‚1.7884684869629269e+09 _runtime‚4540.149377509 _step‚313 & lr-AdamW/pg1‚0.0007103893351387208r&#¶ÌþÆ & lr-AdamW/pg1‚0.0007103893351387208 & lr-AdamW/pg2‚0.0007103893351387208 trainer/global_step‚3849 % _timestamp‚1.7884684869629269e+09 _runtime‚4540.149377509 _step‚313¹>ŠÒ–2ÿ- _wandbruntime‚4540  _runtime‚4540ü¶Gƒß'€ÙO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104725887 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104699543 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164602536225792 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚156951406659584 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7651381237760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27844396 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚174021855744512 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 % _timestamp‚1.7884684869640963e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43804318 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7783076228096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7629457131604 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚164602787897344 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚149336238804688 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73278439 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43804316 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7782791900160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104699543 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73278439 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73278437 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚156951388656640 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚174021547476480 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚149336255844048 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7629686112688 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚164602787897344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚156965941956736 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚700 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚166238755576320 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 _step‚314 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73278437 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 %åˆä'ÞO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73278437 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104725887 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104699543 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164602536225792 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚156951406659584 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7651381237760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27844396 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚174021855744512 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 % _timestamp‚1.7884684869640963e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43804318 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7783076228096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7629457131604 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚164602787897344 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚149336238804688 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73278439 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43804316 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7782791900160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104699543 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73278439 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73278437 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚156951388656640 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚174021547476480 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚149336255844048 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7629686112688 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚164602787897344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚156965941956736 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚700 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚166238755576320 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 _step‚314ºùYñÁ2‚- _wandbruntime‚4541  _runtime‚4541³Šo½ƒäM SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104752605 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 304060416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚166280439546368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280120320 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚149373786201472 trainer/global_step‚3849 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73296714 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚156990692696576 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104752605 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚157005184782702 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71666240 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚156990692696576 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚164644037973504 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178022947 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚157005430803354 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7653345276928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚166280463486464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7653111595008 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178049319 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚164644037973504 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7631644601882 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7653345276928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1740652359»¤Im-35744 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 _runtime‚4541.394953178 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43815118 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73296712 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7784796389376 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104726235 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73296712 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚149373769162112 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178022947 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71665560 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 _step‚315 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27850442 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7785076509696 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27851120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚732967143¨q$ï&„éM EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73296712 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7784796389376 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104726235 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73296712 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚149373769162112 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178022947 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71665560 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 _step‚315 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27850442 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7785076509696 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27851120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73296714 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104752605 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 304060416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚166280439546368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280120320 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚149373786201472 trainer/global_step‚3849 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73296714 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚156990692696576 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104752605 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚157005184782702 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71666240 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚156990692696576 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚164644037973504 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178022947 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚157005430803354 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7653345276928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚166280463486464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7653111595008 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178049319 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚164644037973504 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7631644601882 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7653345276928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚174065235935744 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 _runtime‚4541.394953178 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43815118 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚158940151848000 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7727624153196 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚699 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 180229682 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180256028 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7727395172112 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚158915690506240 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚166665053782016 _step‚318 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74192187 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚176205944339968 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74192187 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74192189 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44344418 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28191543 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚158940397868444 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 trainer/global_step‚3899 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _runtime‚4594.339693127 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7749363275776 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7749363275776 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72535961 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7882966321152 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚176205625586176 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 $ _timestamp‚1.788468541154657e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74192189 TGDeviceStatsMonitor.on_train_batch_start/inactive_spl«Ä¬F~it.large_pool.freed‚44344416 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚168322978018816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚158915708509184 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚158915690506240 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7749596944384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106063839 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚166665053782016 _step‚318 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74192187 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚176205944339968 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74192187 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74192189 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44344418 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28191543 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚158940397868444 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 trainer/global_step‚3899 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _runtime‚4594.339693127 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7749363275776 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7749363275776 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72535961 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7882966321152 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚176205625586176 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 $ _timestamp‚1.788468541154657e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74192189 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44344416 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚168322978018816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚158915708509184 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚158915690506240 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7749596944384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106063839 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚158940151848000 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7727624153196 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚699 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 180229682 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2¾UìB]2–- _wandbruntime‚4595  _runtime‚4595n¡.«ê&—äM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7751327301632 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44355218 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7884658840576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74210464 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚168364638048768 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74210462 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 325031936 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚151250304072672 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7729582642390 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚158954976543232 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180301021 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚151250287033312 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106090557 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106064187 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74210464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884685421655629e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚158954994546176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72552797 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚166706555529728 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7751560983552 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72553504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚158954994546176 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7751560983552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚166706303844864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7884959932416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚176249296889344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚168364661988864 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180274649 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106090557 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚707 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28198284 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚705 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚319 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180274649 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106090557 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚707 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28198284 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚705 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚319 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44355218 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7884658840576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74210464 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚168364638048768 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74210462 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 325031936 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚151250304072672 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7729582642390 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚158954976543232 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180301021 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚151250287033312 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106090557 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106064187 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74210464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884685421655629e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚158954994546176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72552797 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚166706555529728 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7751560983552 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72553504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚158954994546176 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7751560983552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚166706303844864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7884959932416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚176249296889344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚168364661988864 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107375447 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75105937 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚694 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚178389921128960 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚160914853780152 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 $ _timestamp‚1.788468596527626e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 182507730 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚168727823009792 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75105939 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚160879992355840 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚178389610763776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75105937 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107401791 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚170407176521216 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7847812651008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107401791 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚160880010358784 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7847578982400 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚160880010358784 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚153089274547088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107375447 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚692 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7825333212620 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚153089291586448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7982744607744 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28538243 _runtime‚4649.712655636 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 trainer/global_step‚3949 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚160879992355840 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7847578982400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44884518 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚160914607759708 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 182481384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73422761 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚168727571338240 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚170407152581120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7825562193704 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚168727823009792 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73422067 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚322 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168727571338240 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28537551 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75105939 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44884516 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006Ôï#nä'§ÞO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75105937 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107401791 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚170407176521216 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7847812651008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107401791 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚160880010358784 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7847578982400 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚160880010358784 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚153089274547088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107375447 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚692 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7825333212620 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚153089291586448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7982744607744 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28538243 _runtime‚4649.712655636 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 trainer/global_step‚3949 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚160879992355840 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7847578982400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44884518 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚160914607759708 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 182481384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73422761 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚168727571338240 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚170407152581120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7825562193704 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚168727823009792 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73422067 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚322 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168727571338240 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28537551 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75105939 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44884516 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7982458182656 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107375447 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75105937 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚694 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚178389921128960 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚160914853780152 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 $ _timestamp‚1.788468596527626e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 182507730 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚168727823009792 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75105939 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚160879992355840 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚178389610763776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528ˆ¹Ò‘¨‚(:† ôºçÔ¶´ô‡ gpu.0.gpu‚28.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚140.754' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.1885 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ¯›p©‚(:å ôºçÔˆ¸Äˆ disk.sdb.in‚ 3.49609375 disk.sdb1.out‚0 proc.memory.rssMB‚3030.25 disk.sda2.out‚ 505.337890625 disk.sdb2.out‚ 505.337890625 disk.rbd2.out‚ 1.341796875 disk.md0.out‚ 487.49609375 disk.nvme1n1.in‚ 200395.53125 disk.sdb.out‚ 505.337890625 network.recv‚ 1.570581e+06% disk./.usageGB‚2974.7525024414062 disk.md0.in‚ 7.12109375# disk.nvme1n1.out‚40163.55859375 disk.sda.in‚3.625# disk.nvme0n1.out‚19933.63671875 disk.sda1.in‚0 disk.sdb2.in‚ 3.49609375 disk.rbd0.in‚0.015625 network.sent‚ 9.075788e+06 disk.dm-0.in‚528701.00390625 disk.sda2.in‚3.625$ memory_percent‚34.97750759124756) proc.memory.percent‚18.49517822265625 disk.sda1.out‚0 disk.md0p1.out‚ 487.49609375 disk.sdb1.in‚0 proc.cpu.threads‚17( disk./.usagePercent‚83.1753249989078 disk.dm-0.out‚ 60097.1953125 disk.md0p1.in‚ 7.12109375 disk.rbd2.in‚0* proc.memory.availableMB‚10653.28515625 cpu‚25.15601391144894 disk.sda.out‚ 505.337890625# disk.nvme0n1.in‚328305.47265625 disk.rbd0.out‚ 1.2783203125J ‡%2ª- _wandbruntime‚4650  _runtime‚4650ƒEþ¯ «äM PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182552723 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7984742668288 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 182552723 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚706 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚160919296395776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 182526351 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7827520682898 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107428509 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28545087 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107402139 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚3949 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 H@DeviceStatsMonito!ˆ«ÿàr.on_train_batch_end/inactive_split.all.current‚708 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75124214 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7849543008256 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚160919278392832 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7849543008256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107428509 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚160919296395776 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚178433603159552 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75124212 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚160919278392832 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7849776690176 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44895320 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107402139 % _timestamp‚1.7884685974818578e+09 _runtime‚4650.667435252 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚168769073085952 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73439699 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚168769073085952 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚153126821943872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚168768821401088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚153126804904512 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 182526351 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚178433286516224 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚170448860491264 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7984449965056ëK¾Sï&¬éM % _timestamp‚1.7884685974818578e+09 _runtime‚4650.667435252 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚168769073085952 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73439699 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚168769073085952 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚153126821943872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚168768821401088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚153126804904512 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 182526351 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚178433286516224 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚170448860491264 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7984449965056 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182552723 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7984742668288 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 182552723 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚706 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚160919296395776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 182526351 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7827520682898 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107428509 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28545087 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107402139 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚3949 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚708 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75124214 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7849543008256 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚160919278392832 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7849543008256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107428509 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚160919296395776 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚178433603159552 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75124212 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚160919278392832 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7849776690176 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44895320 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107402139ñáí?2­- _wandbruntime‚4650  _runtime‚4650äËý\®Ö 0loss/FBt3/assignment_loss‚0.0484916889298585 1loss/FBt4/assignment_loss‚0.09376452373714983 .loss/FRt1/detection_loss‚2.785095816478133 /loss/FBt2/detection_loss‚2.2589403366049132 epoch‚3 trainer/global_step‚3949 _runtime‚4650.669303541 1loss/SRqqt4/detection_loss‚2.3548081243360364 .loss/FBt1/detection_loss‚3.935452019174894 _step‚324 0loss/FRt3/assignment_loss‚1.5953711946055573 0loss/FRt4/assignment_loss‚2.8924508605684554 1loss/SRqqt2/assignment_loss‚2.324734759243096 % _timestamp‚1.7884685974845974e+09 /loss/FRt1/assignment_loss‚2.888496670871973 /loss/FRt2/detection_loss‚1.9123693649453344 /loss/FRt3/detection_loss‚1.6730627165931575 /loss/FRt4/detection_loss‚3.1932607917558578 1loss/SRqqt3/detection_loss‚0.9814259727795919 1loss/SRqqt2/detection_loss‚2.4435734959209667 2loss/SRqqt1/assignment_loss‚1.7491656050414723 2loss/SRqqt3/assignment_loss‚0.9045981174532951 1loss/SRqqt4/assignment_loss‚1.689247740684329 1loss/FBt2/assignment_loss‚0.12666379881557077 /loss/FBt3/detection_loss‚0.8659067140133293 .loss/FBt4/detection_loss‚3.055248586460948 1loss/SRqqt1/detection_loss‚1.7595195361212188 %loss/total_loss‚45.07379662767579 0loss/FRt2/assignment_loss‚3.3600646363837376 1loss/FBt1/assignment_loss‚0.18208355615691593'ãa¯Û 0loss/FRt2/assignment_loss‚3.3600646363837376 1loss/FBt1/assignment_loss‚0.18208355615691593 0loss/FBt3/assignment_loss‚0.0484916889298585 1loss/FBt4/assignment_loss‚0.09376452373714983 .loss/FRt1/detection_loss‚2.785095816478133 /loss/FBt2/detection_loss‚2.2589403366049132 epoch‚3 trainer/global_step‚3949 _runtime‚4650.669303541 1loss/SRqqt4/detection_loss‚2.3548081243360364 .loss/FBt1/detection_loss‚3.935452019174894 _step‚324 0loss/FRt3/assignment_loss‚1.5953711946055573 0loss/FRt4/assignment_loss‚2.8924508605684554 1loss/SRqqt2/assignment_loss‚2.324734759243096 % _timestamp‚1.7884685974845974e+09 /loss/FRt1/assignment_loss‚2.888496670871973 /loss/FRt2/detection_loss‚1.9123693649453344 /loss/FRt3/detection_loss‚1.6730627165931575 /loss/FRt4/detection_loss‚3.1932607917558578 1loss/SRqqt3/detection_loss‚0.9814259727795919 1loss/SRqqt2/detection_loss‚2.4435734959209667 2loss/SRqqt1/assignment_loss‚1.7491656050414723 2loss/SRqqt3/assignment_loss‚0.9045981174532951 1loss/SRqqt4/assignment_loss‚1.689247740684329 1loss/FBt2/assignment_loss‚0.12666379881557077 /loss/FBt3/detection_loss‚0.8659067140133293 .loss/FBt4/detection_loss‚3.055248586460948 1loss/SRqqt1/detection_loss‚1.7595195361212188 %loss/total_loss‚45.07379662767579Ä^¤Ôd°‚(:… ƒ»çÔ¥Õ† gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚135.184' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.796 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òSEäv±‚(:ë ƒ»çÔ‘ш‡ disk.sda2.in‚3.625) proc.memory.percent‚18.49517822265625 network.sent‚ 9.171167e+06 disk.md0p1.out‚ 489.046875! disk.sdb2.out‚506.95849609375 proc.memory.rssMB‚3030.25" disk.nvme0n1.out‚ 20201.8046875 disk.sda1.in‚0 disk.rbd0.out‚ 1.34130859375 disk.sdb.in‚ 3.49609375 disk.sdb.out‚506.95849609375 network.recv‚ 1.576471e+06% disk./.usageGB‚2974.8193855285645# disk.nvme0n1.in‚328305.47265625! disk.sda2.out‚506.95849609375 disk.nvme1n1.in‚ 200395.53125 disk.sdb1.out‚0 disk.dm-0.out‚60365.51953125 disk.md0p1.in‚ 7.12109375 disk.rbd2.out‚ 1.341796875* proc.memory.availableMB‚10653.45703125 disk.dm-0.in‚528701.00390625 disk.sdb2.in‚ 3.49609375 disk.md0.out‚ 489.046875# disk.nvme1n1.out‚40163.71484375) disk./.usagePercent‚83.17719507801749 disk.sda1.out‚0 disk.md0.in‚ 7.12109375 disk.rbd2.in‚0$ memory_percent‚34.97645854949951 disk.sdb1.in‚0 disk.rbd0.in‚0.015625 cpu‚25.15563871862434 proc.cpu.threads‚17 disk.sda.in‚3.625 disk.sda.out‚506.95849609375LÌù›²‚(: ’»çÔÃÔ͇ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚117.81' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.452499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à.ƒoo³‚(:ä ’»çÔò“ñˆ disk.dm-0.out‚60529.25! disk.sda2.out‚508.28173828125 disk.sdb.in‚ 3.49609375% memory_percent‚34.977054595947266( proc.memory.availableMB‚ 10653.359375 proc.cpu.threads‚17 network.sent‚ 9.27065e+06! disk.nvme0n1.out‚ 20365.453125 disk.md0p1.out‚ 490.30859375! disk.sdb2.out‚508.28173828125 network.recv‚ 1.583354e+06 disk.sda1.in‚0 disk.md0.in‚ 7.12109375 disk.md0.out‚ 490.30859375 disk.sda.in‚3.625 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 1.34130859375 proc.memory.rssMB‚3030.25 cpu‚25.155282337186648 disk.sdb1.out‚0# disk.nvme0n1.in‚328305.47265625 disk.rbd2.in‚0 disk.rbd2.out‚ 1.341796875 disk.nvme1n1.in‚ 200395.53125 disk.sdb.out‚508.28173828125) disk./.usagePercent‚83.18075743312981$ disk./.usageGB‚2974.946792602539 disk.dm-0.in‚528701.00390625 disk.sda2.in‚3.625 disk.sdb2.in‚ 3.49609375 disk.sda.out‚508.28173828125 disk.md0p1.in‚ 7.12109375! disk.nvme1n1.out‚ 40163.796875 disk.sda1.out‚0 disk.sdb1.in‚0) proc.memory.percent‚18.49517822265625¿Ý®þ‘´‚(:† ¡»çÔ܇‘‡ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚140.478' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.1195 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F! gµ‚(:Ü ¡»çÔ¡Ìþ‡ disk.rbd2.out‚ 1.341796875 disk.md0p1.in‚ 7.12109375 disk.rbd0.out‚ 1.404296875# disk.nvme0n1.out‚20597.26171875 disk.sda1.in‚0$ disk./.usageGB‚2975.071918487549 disk.sda2.in‚3.625 disk.nvme1n1.in‚ 200395.53125 disk.sdb1.in‚0 network.recv‚ 1.58786e+06 disk.sda2.out‚ 510.28515625 disk.md0p1.out‚ 492.2578125 disk.md0.in‚ 7.12109375 disk.sda.in‚3.625 disk.dm-0.out‚ 60761.28125 disk.sda1.out‚0 cpu‚25.15490529812927 disk.md0.out‚ 492.2578125# disk.nvme1n1.out‚40164.01953125 disk.sdb.in‚ 3.49609375 disk.sdb1.out‚0# disk.nvme0n1.in‚328305.47265625% memory_percent‚34.977173805236816 network.sent‚ 9.276564e+06 disk.sdb2.out‚ 510.28515625 disk.sdb.out‚ 510.28515625 disk.sda.out‚ 510.28515625 disk.dm-0.in‚528701.00390625 proc.memory.rssMB‚3030.25) proc.memory.percent‚18.49517822265625 disk.sdb2.in‚ 3.49609375* proc.memory.availableMB‚10653.33984375 proc.cpu.threads‚17) disk./.usagePercent‚83.18425600524394 disk.rbd0.in‚0.015625 disk.rbd2.in‚0quã2¶- _wandbruntime‚4704  _runtime‚4704 ÒÖçÇ·Á _runtime‚4704.967854502 _step‚325 & lr-AdamW/pg1‚0.0006977855836178171 & lr-AdamW/pg2‚0.0006977855836178171 trainer/global_step‚3999 % _timestamp‚1.7884686517827942e+09ÒcUÌ¸Æ & lr-AdamW/pg1‚0.0006977855836178171 & lr-AdamW/pg2‚0.0006977855836178171 trainer/global_step‚3999 % _timestamp‚1.7884686517827942e+09 _runtime‚4704.967854502 _step‚325ŔٟX2¹- _wandbruntime‚4704  _runtime‚4704€›w€Ó ºÚO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚172491351083520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108713399 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 _runtime‚4704.969444379 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚170790340566016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74309493 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 _step‚326 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚170790088894464 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76019689 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76019687 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚699 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7923500234212 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7945794689024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74310194 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45424616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚170790340566016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚701 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚180574072464896 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76019687 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7946028357632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108739743 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7946028357632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8082421501952 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚162889063671416 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108713399 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚154965809457648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚172491375023616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚162889309691860 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76019689 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚180573772585472 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275939328 B peak‚ 6236507136 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28884877 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8082697441280 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108739743 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299879424 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 184733086 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚162844312208384 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28885576 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚162844312208384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45424618 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚154965792418288™¢å'»ßO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚170790340566016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚701 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚180574072464896 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76019687 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7946028357632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108739743 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7946028357632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8082421501952 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚162889063671416 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108713399 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚154965809457648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚172491375023616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚162889309691860 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76019689 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚180573772585472 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275939328 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 184733086 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚162844312208384 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28885576 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚162844312208384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45424618 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚154965792418288 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚172491351083520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108713399 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 _runtime‚4704.969444379 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚170790340566016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74309493 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 _step‚326 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚170790088894464 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76019689 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76019687 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚699 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7923500234212 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7945794689024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74310194 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45424616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0ƨWô£2¼- _wandbruntime‚4706  _runtime‚4706Âüáê&½äM F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚4706.070013802 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚170831338957312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28892529 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45435420 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76037962 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7947758714880 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚180617763348992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚155003322775712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚162883598245376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45435418 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7947992396800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚180617450899968 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7925458723406 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 184778053 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚172533058993664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108766461 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7947758714880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚170831590642176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 184778053 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 184804425 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108740091 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7925229742114 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚162883580242432 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚162928798538478 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 trainer/global_step‚3999 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚170831590642176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184804425 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108740091 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚162883598245376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚172533035053568 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74327269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7947992396800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7925458723406 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 184778053 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚172533058993664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108766461 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7947758714880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚170831590642176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 184778053 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 184804425 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108740091 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7925229742114 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚162883580242432 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚162928798538478 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 trainer/global_step‚3999 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚170831590642176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184804425 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108740091 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚162883598245376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚172533035053568 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74327269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7947992396800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ÇÜ mé2¿- _wandbruntime‚4706  _runtime‚47062zÌ ÀÛ epoch‚3 trainer/global_step‚3999 /loss/FRt3/detection_loss‚1.2433323185168095 1loss/SRqqt1/detection_loss‚2.0155517951301904 0loss/FBt3/detection_loss‚0.37851231884477393 0loss/FRt1/assignment_loss‚2.3873265578466305 0loss/FRt3/assignment_loss‚1.0704530166156256 1loss/FBt3/assignment_loss‚0.04562966699771615 1loss/SRqqt3/detection_loss‚0.7992758661116425 _step‚328 0loss/FRt4/assignment_loss‚2.5523127992947896 2loss/SRqqt1/assignment_loss‚1.4818153776552365 /loss/FRt4/detection_loss‚2.8325100501378375 /loss/FBt2/detection_loss‚2.5061287513146033 /loss/FRt2/assignment_loss‚4.031880675421821 2los7'L7Ás/SRqqt2/assignment_loss‚1.4320809329892028 1loss/FBt4/assignment_loss‚0.14612436225359654 2loss/SRqqt4/assignment_loss‚1.6859619779846606 2loss/FBt2/assignment_loss‚0.033196213642668226 -loss/FRt2/detection_loss‚2.29482316541382 1loss/SRqqt2/detection_loss‚2.6955205530955872 1loss/SRqqt4/detection_loss‚2.1700598200162253 /loss/FRt1/detection_loss‚2.6797157210462235 .loss/FBt4/detection_loss‚2.204550949009982 &loss/total_loss‚41.106729651562816 2loss/SRqqt3/assignment_loss‚0.8941500679638825 0loss/FBt1/assignment_loss‚0.2274266733325021 /loss/FBt1/detection_loss‚3.2983900209267936 % _timestamp‚1.7884686528875597e+09 _runtime‚4706.072475103í7 ™fÁà /loss/FRt4/detection_loss‚2.8325100501378375 /loss/FBt2/detection_loss‚2.5061287513146033 /loss/FRt2/assignment_loss‚4.031880675421821 2loss/SRqqt2/assignment_loss‚1.4320809329892028 1loss/FBt4/assignment_loss‚0.14612436225359654 2loss/SRqqt4/assignment_loss‚1.6859619779846606 2loss/FBt2/assignment_loss‚0.033196213642668226 -loss/FRt2/detection_loss‚2.29482316541382 1loss/SRqqt2/detection_loss‚2.6955205530955872 1loss/SRqqt4/detection_loss‚2.1700598200162253 /loss/FRt1/detection_loss‚2.6797157210462235 .loss/FBt4/detection_loss‚2.204550949009982 &loss/total_loss‚41.106729651562816 2loss/SRqqt3/assignment_loss‚0.8941500679638825 0loss/FBt1/assignment_loss‚0.2274266733325021 /loss/FBt1/detection_loss‚3.2983900209267936 % _timestamp‚1.7884686528875597e+09 _runtime‚4706.072475103 epoch‚3 trainer/global_step‚3999 /loss/FRt3/detection_loss‚1.2433323185168095 1loss/SRqqt1/detection_loss‚2.0155517951301904 0loss/FBt3/detection_loss‚0.37851231884477393 0loss/FRt1/assignment_loss‚2.3873265578466305 0loss/FRt3/assignment_loss‚1.0704530166156256 1loss/FBt3/assignment_loss‚0.04562966699771615 1loss/SRqqt3/detection_loss‚0.7992758661116425 _step‚328 0loss/FRt4/assignment_loss‚2.5523127992947896 2loss/SRqqt1/assignment_loss‚1.4818153776552365ÈàW`Èm‚(:â °»çÔøŒö‡$ disk./.usageGB‚2975.071403503418 disk.md0.in‚ 7.12109375 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 1.404296875 disk.sdb1.out‚0# disk.nvme0n1.in‚328305.47265625 disk.dm-0.out‚61021.5 disk.sda2.in‚3.625 disk.sdb.out‚511.71240234375! disk.sda2.out‚511.71240234375 network.recv‚ 1.592112e+06! disk.sdb2.out‚511.71240234375 proc.cpu.threads‚17 disk.nvme1n1.in‚ 200395.53125# disk.nvme1n1.out‚40164.19140625# disk.nvme0n1.out‚20857.30859375 disk.sda1.in‚0 cpu‚25.1544689557537 disk.sdb2.in‚ 3.49609375 disk.sdb1.in‚0 disk.dm-0.in‚528701.00390625 disk.rbd2.in‚0 disk.rbd2.out‚ 1.341796875* proc.memory.availableMB‚10653.23828125 proc.memory.rssMB‚3030.25 disk.md0.out‚ 493.6328125 disk.sdb.in‚ 3.49609375 disk.sda.out‚511.71240234375 disk.md0p1.in‚ 7.12109375$ memory_percent‚34.97779369354248) proc.memory.percent‚18.49517822265625 network.sent‚ 9.280405e+06( disk./.usagePercent‚83.1842416060721 disk.sda.in‚3.625 disk.sda1.out‚0 disk.md0p1.out‚ 493.6328125”ÚéœÃ‚(:‘ °»çÔ¶ò®ˆ gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚119.978' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.994499999999995 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jXjÂŽÄ‚(:ƒ ¿»çԱ† gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚137.36' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.34 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03YqžhÅ‚(:Ý ¿»çÔ‡óˆ disk.rbd0.out‚ 1.404296875$ memory_percent‚34.97757911682129( disk./.usagePercent‚83.1842416060721$ disk./.usageGB‚2975.071403503418 disk.nvme1n1.out‚ 40164.3125 disk.sda1.in‚0) proc.memory.percent‚18.49517822265625 disk.rbd2.in‚0 disk.sda.in‚3.625# disk.nvme0n1.in‚328305.47265625 disk.dm-0.in‚528701.00390625 disk.sda2.in‚3.625 disk.md0p1.in‚ 7.12109375 disk.rbd2.out‚ 1.341796875 disk.sdb1.in‚0 disk.sda.out‚ 513.173828125) proc.memory.availableMB‚ 10653.2734375 proc.memory.rssMB‚3030.25 disk.sdb.in‚ 3.49609375 disk.sdb1.out‚0 disk.dm-0.out‚61373.68359375 proc.cpu.threads‚17 network.sent‚ 9.387075e+06 disk.sdb2.in‚ 3.49609375 disk.nvme1n1.in‚ 200395.53125 disk.sdb.out‚ 513.173828125 disk.md0p1.out‚ 495.0390625 cpu‚25.15408611984386 disk.sda2.out‚ 513.173828125 disk.sdb2.out‚ 513.173828125 disk.rbd0.in‚0.015625# disk.nvme0n1.out‚21209.37109375 disk.sda1.out‚0 network.recv‚ 1.598305e+06 disk.md0.in‚ 7.12109375 disk.md0.out‚ 495.0390625©Å–‘Æ‚(:† λçÔ‘­à… gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚132.122' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.0305 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wME`Ç‚(:Õ λçÔÔ»‡# disk.nvme0n1.in‚328305.47265625) proc.memory.percent‚18.49517822265625 disk.sdb1.out‚0 disk.sda.in‚3.625 disk.md0p1.in‚ 7.12109375 disk.md0p1.out‚ 496.984375 cpu‚25.15366395078315% disk./.usageGB‚2973.7389373779297 disk.sdb2.out‚ 515.171875 disk.md0.in‚ 7.12109375 disk.md0.out‚ 496.984375 disk.rbd2.out‚ 1.341796875 disk.nvme1n1.in‚ 200395.53125 disk.sdb.in‚ 3.49609375$ memory_percent‚34.97755527496338 disk.sda1.out‚0 disk.sda2.out‚ 515.171875# disk.nvme1n1.out‚40164.40234375 proc.cpu.threads‚17 disk.sdb.out‚ 515.171875 disk.dm-0.in‚528701.00390625 disk.sda2.in‚3.625 disk.rbd0.out‚ 1.404296875 disk.sda1.in‚0 network.recv‚ 1.602935e+06) disk./.usagePercent‚83.14698529552305 network.sent‚ 9.395596e+06 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.sdb1.in‚0" disk.nvme0n1.out‚ 21364.1953125 disk.dm-0.out‚61528.59765625* proc.memory.availableMB‚10653.27734375 proc.memory.rssMB‚3030.25 disk.sdb2.in‚ 3.49609375 disk.sda.out‚ 515.171875£¦‘È‚(:† Ý»çÔþ…ч gpu.0.gpu‚5.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚144.821' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.20525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(ÊDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚164863519583124 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45964716 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8021209293636 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 186984788 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚164808596055040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76933439 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8182468087808 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚156842327328848 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚172852606450688 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚164808614057984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚172852606450688 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110077695 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚156842310289488 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚172852858122240 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75196277 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚174575549585920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚676dy¦Ìå'ÏßO F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚172852858122240 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75196277 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚174575549585920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚676 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76933437 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110051351 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚164863519583124 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45964716 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8021209293636 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 186984788 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚164808596055040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76933439 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8182468087808 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚156842327328848 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚172852606450688 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚164808614057984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚172852606450688 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110077695 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚156842310289488 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚172894108198400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚164847900094976 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚164847900094976 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75213306 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 _step‚331 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110078043 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.Ÿ%ú[lsmall_pool.freed‚ 8184187038208 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8023396763914 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76951714 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8046208103424 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚164903254450186 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 187029755 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76951712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚182801726751744 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8184469255680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚182801420594176 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884687055018022e+09 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8046208103424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187056127 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76951714 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚174617257496064 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚4758.687390943 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚164847882092032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚650 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45975520 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110104413 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚164847882092032 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚174617233555968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8045974421504 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110078043 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29237788 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45975518 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 187056127 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚172894108198400 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 _step‚331 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110078043 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8184187038208 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8023396763914 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76951714 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8046208103424 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚164903254450186 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 187029755 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76951712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚182801726751744 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8184469255680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚182801420594176 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884687055018022e+09 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8046208103424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187056127 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76951714 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚174617257496064 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚4758.687390943 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚164847882092032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚650 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45975520 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110104413 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚164847882092032 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚174617233555968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8045974421504 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110078043 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29237788 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45975518 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 187056127 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚172894108198400 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚172894108198400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚164847900094976 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚164847900094976 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75213306 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0ËÁ+ó2Ó- _wandbruntime‚4758  _runtime‚4758ÂFâB]Ô× 2loss/SRqqt3/assignment_loss‚0.8163344600876228 /loss/FBt3/detection_loss‚0.5767130706685059 trainer/global_step‚4049 1loss/SRqqt3/detection_loss‚1.0114066455973667 /loss/FBt2/detection_loss‚1.9342503945032756 _runtime‚4758.688937781 2loss/SRqqt1/assignment_loss‚1.4689312331816728 0loss/FBt2/assignment_loss‚0.3005559059640897 1loss/SRqqt1/detection_loss‚2.3849752212271973 /loss/FBt1/detection_loss‚2.2869761098514902 /loss/FBt4/detection_loss‚2.9110230604807534 0loss/FBt1/assignment_loss‚0.2007344504812119 0loss/SRqqt2/detection_loss‚2.235701460992136 0loss/FRt2/assignment_loss‚2.9203030625979105 2loss/SRqqt2/assignment_loss‚1.4625864956648118 .loss/FRt2/detection_loss‚2.194118961121421 0loss/SRqqt4/detection_loss‚2.796753074114139 0loss/FRt3/assignment_loss‚1.2412938354890557 .loss/FRt1/detection_loss‚2.901283767716638 epoch‚3 _step‚332 2loss/FBt4/assignment_loss‚0.017283233166494938 /loss/FRt3/detection_loss‚1.5726134316830696 % _timestamp‚1.7884687055042384e+09 2loss/SRqqt4/assignment_loss‚1.0712131726054044 1loss/FBt3/assignment_loss‚0.06839000963187249 /loss/FRt4/detection_loss‚3.3159141919829627 %loss/total_loss‚41.95311696333035 0loss/FRt1/assignment_loss‚2.7363931310587915 /loss/FRt4/assignment_loss‚3.527368583462455iËbÕÜ 2loss/SRqqt1/assignment_loss‚1.4689312331816728 0loss/FBt2/assignment_loss‚0.3005559059640897 1loss/SRqqt1/detection_loss‚2.3849752212271973 /loss/FBt1/detection_loss‚2.2869761098514902 /loss/FBt4/detection_loss‚2.9110230604807534 0loss/FBt1/assignment_loss‚0.2007344504812119 0loss/SRqqt2/detection_loss‚2.235701460992136 0loss/FRt2/assignment_loss‚2.9203030625979105 2loss/SRqqt2/assignment_loss‚1.4625864956648118 .loss/FRt2/detection_loss‚2.194118961121421 0loss/SRqqt4/detection_loss‚2.796753074114139 0loss/FRt3/assignment_loss‚1.2412938354890557 .loss/FRt1/detection_loss‚2.901283767716638 epoch‚3 _step‚332 2loss/FBt4/assignment_loss‚0.017283233166494938 /loss/FRt3/detection_loss‚1.5726134316830696 % _timestamp‚1.7884687055042384e+09 2loss/SRqqt4/assignment_loss‚1.0712131726054044 1loss/FBt3/assignment_loss‚0.06839000963187249 /loss/FRt4/detection_loss‚3.3159141919829627 %loss/total_loss‚41.95311696333035 0loss/FRt1/assignment_loss‚2.7363931310587915 /loss/FRt4/assignment_loss‚3.527368583462455 2loss/SRqqt3/assignment_loss‚0.8163344600876228 /loss/FBt3/detection_loss‚0.5767130706685059 trainer/global_step‚4049 1loss/SRqqt3/detection_loss‚1.0114066455973667 /loss/FBt2/detection_loss‚1.9342503945032756 _runtime‚4758.688937781Ì–}x%œÖ‚(:‘ ì»çԲˤ‡ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íÄ2>oׂ(:ä ì»çÔփLj disk.sda.in‚3.625 disk.md0p1.out‚ 499.734375 disk.sdb2.in‚ 3.49609375 cpu‚25.152792681076022 disk.md0.in‚ 7.12109375 disk.rbd0.out‚ 1.404296875 network.sent‚ 9.418067e+06 network.recv‚ 1.611648e+06 disk.nvme1n1.out‚ 40164.78125 disk.dm-0.in‚528701.00390625 disk.sda1.in‚0$ memory_percent‚34.97776985168457# disk.nvme0n1.in‚328305.47265625! disk.sdb2.out‚518.05322265625 disk.rbd2.out‚ 1.341796875) proc.memory.availableMB‚ 10653.2421875$ disk./.usageGB‚2973.694980621338 disk.sdb1.in‚0! disk.sda2.out‚518.05322265625) disk./.usagePercent‚83.14575624621142 disk.dm-0.out‚ 61529.7421875 disk.md0.out‚ 499.734375 disk.sda1.out‚0 disk.sda2.in‚3.625 disk.md0p1.in‚ 7.12109375 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 proc.memory.rssMB‚3030.25) proc.memory.percent‚18.49517822265625 proc.cpu.threads‚17 disk.sda.out‚518.05322265625" disk.nvme0n1.out‚ 21364.9609375 disk.sdb.in‚ 3.49609375 disk.sdb.out‚518.05322265625 disk.sdb1.out‚0 disk.nvme1n1.in‚ 200395.53125ž©µ’Ø‚(:‡ û»çÔºõŽ† gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚138.919' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.72975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ñ@wÙ‚(:ì û»çÔæÇ‘‡ disk.nvme1n1.in‚ 200395.53125 disk.sda.in‚3.625! disk.sdb2.out‚519.97802734375 disk.md0.out‚ 501.59765625 cpu‚25.152436297036004) proc.memory.percent‚18.49517822265625 network.sent‚ 9.51206e+06% disk./.usageGB‚2973.6483459472656 disk.sdb1.in‚0 disk.md0p1.out‚ 501.59765625 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 1.404296875 proc.cpu.threads‚17) disk./.usagePercent‚83.14445232120623 disk.sdb.in‚ 3.49609375# disk.nvme0n1.out‚21437.24609375 disk.sda1.in‚0 disk.rbd2.out‚ 1.341796875$ memory_percent‚34.97779369354248! disk.nvme1n1.out‚ 40165.171875 disk.sdb1.out‚0 disk.sda.out‚519.97802734375 disk.dm-0.out‚61602.41796875! disk.sda2.out‚519.97802734375 disk.md0p1.in‚ 7.12109375 disk.rbd2.in‚0 proc.memory.rssMB‚3030.25# disk.nvme0n1.in‚328305.47265625 network.recv‚ 1.618056e+06 disk.sdb.out‚519.97802734375 disk.dm-0.in‚528701.00390625 disk.sda1.out‚0 disk.sda2.in‚3.625 disk.sdb2.in‚ 3.49609375 disk.md0.in‚ 7.12109375* proc.memory.availableMB‚10653.23828125Òšòä‘Ú‚(:† Š¼çÔãÔ×… gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚134.298' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.5745 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&³‹qÛ‚(:æ Š¼çÔ„Ô͇ network.sent‚ 9.520698e+06 network.recv‚ 1.622734e+06 disk.sdb2.in‚ 3.49609375" disk.nvme1n1.out‚ 40165.2734375 disk.sdb1.out‚0 proc.memory.rssMB‚3030.25# disk.nvme0n1.out‚21574.08984375 disk.md0.out‚ 503.2109375 disk.rbd2.out‚ 1.341796875 disk.dm-0.out‚61739.36328125 disk.md0p1.in‚ 7.12109375 disk.sdb2.out‚521.6591796875 disk.md0.in‚ 7.12109375 disk.dm-0.in‚528701.00390625 disk.rbd0.in‚0.015625) proc.memory.percent‚18.49517822265625 disk.rbd2.in‚0 disk.sdb.out‚521.6591796875 disk.sda.in‚3.625 cpu‚25.152009828082882% disk./.usageGB‚2973.8573989868164 disk.sda1.out‚0 disk.sda2.in‚3.625 disk.nvme1n1.in‚ 200395.53125 disk.sdb1.in‚0$ memory_percent‚34.97934341430664) disk./.usagePercent‚83.15029753168757 disk.md0p1.out‚ 503.2109375 disk.rbd0.out‚ 1.404296875 disk.sdb.in‚ 3.49609375( proc.memory.availableMB‚ 10652.984375# disk.nvme0n1.in‚328305.47265625 disk.sda1.in‚0 disk.sda2.out‚521.6591796875 disk.sda.out‚521.6591796875 proc.cpu.threads‚17fGÑz2Ü- _wandbruntime‚4810  _runtime‚4810þ¦™ÇÝÁ % _timestamp‚1.7884687573978977e+09 _runtime‚4810.583055895 _step‚333 & lr-AdamW/pg1‚0.0006890840725132828 & lr-AdamW/pg2‚0.0006890840725132828 trainer/global_step‚4099uþçMÌÞÆ trainer/global_step‚4099 % _timestamp‚1.7884687573978977e+09 _runtime‚4810.583055895 _step‚333 & lr-AdamW/pg1‚0.0006890840725132828 & lr-AdamW/pg2‚0.0006890840725132828Í0H02ß- _wandbruntime‚4810  _runtime‚4810½„•}7 àÙO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚174915375678464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77847189 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚166772897904640 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46504816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29577939 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚176659772028416 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77847187 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76082757 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8142226102272 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚158718828160688 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77847189 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111389303 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 trainer/global_step‚4099 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8282355386880 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚166837975494832 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚174915124006912 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 189262836 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8119147334144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8119376315228 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76082071 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111415647 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool]삨.allocated‚ 8142459770880 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚686 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46504818 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325045248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _step‚334 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚166838221515276 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚158718845200048 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111415647 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚166772915907584 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚166772915907584 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136ÿ2ä'áÞO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8142226102272 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8142459770880 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8282054281728 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77847187 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76082757 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8142226102272 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚158718828160688 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77847189 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111389303 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 trainer/global_step‚4099 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8282355386880 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚166837975494832 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚174915124006912 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 189262836 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8119147334144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8119376315228 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76082071 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111415647 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8142459770880 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚686 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46504818 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325045248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _step‚334 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚166838221515276 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚158718845200048 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111415647 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚166772915907584 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚166772915907584 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚174915375678464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77847189 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚166772897904640 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46504816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29577939 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚176659772028416 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77865462 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚166812183941632 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111415995 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚663 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 % _timestamp‚1.7884687585964339e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚184985485881344 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚158756358518112 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 189281457 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚166812183941632 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46515620 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111442365 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76100698 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77865464 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76100035 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 189281457 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚158756375557472 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8144190128128 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77865462 trainer/global_step‚4099 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚176701432058368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8121105823130 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚174956625754624 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 _runtime‚4811.781914946 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189307829 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46515618 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111415995 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8144190128128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29585078 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚166877464341242 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8284338137600 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚174956625754624 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚174956374069760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8284053822976 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29584417 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚166877710361894 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 _step‚335âLï&äéM EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 % _timestamp‚1.7884687585964339e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚184985485881344 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚158756358518112 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 189281457 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚166812183941632 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46515620 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111442365 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76100698 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77865464 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76100035 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 189281457 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚158756375557472 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8144190128128 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77865462 trainer/global_step‚4099 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚176701432058368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8121105823130 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚174956625754624 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 _runtime‚4811.781914946 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189307829 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46515618 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111415995 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8144190128128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29585078 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚166877464341242 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8284338137600 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚174956625754624 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚174956374069760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8284053822976 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29584417 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚166877710361894 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 _step‚335 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77865462 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚166812183941632 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111415995 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚663Ï}l¿42å- _wandbruntime‚4811  _runtime‚4811EV®æÖ /loss/FRt2/assignment_loss‚3.418939377399201 1loss/FBt2/assignment_loss‚0.19104851001539203 1loss/FBt3/assignment_loss‚0.05988144868176156 /loss/FBt4/detection_loss‚1.9665325113705225 trainer/global_step‚4099 % _timestamp‚1.7884687585990407e+09 .loss/FRt1/detection_loss‚2.555358113213019 _step‚336 /loss/FRt4/assignment_loss‚2.733195667465528 1loss/FBt4/assignment_loss‚0.18315755160827912 .loss/FRt4/detection_loss‚2.970693210760752 /loss/FBt3/detection_loss‚0è$ÌV.5843703239161062 epoch‚3 2loss/SRqqt2/assignment_loss‚1.3924909176365021 2loss/SRqqt4/assignment_loss‚2.4585830776587776 /loss/FBt1/detection_loss‚2.9172587022185326 .loss/FBt2/detection_loss‚2.021289326250553 _runtime‚4811.783827035 1loss/SRqqt1/detection_loss‚2.0135734067406763 0loss/SRqqt2/detection_loss‚2.357285024658326 1loss/SRqqt3/detection_loss‚1.0077766973252704 0loss/FRt3/assignment_loss‚1.1536198216424862 2loss/SRqqt1/assignment_loss‚1.2398282549408979 2loss/SRqqt3/assignment_loss‚1.0173504547238714 0loss/FBt1/assignment_loss‚0.5232646361691877 /loss/FRt2/detection_loss‚2.3044414508033872 /loss/FRt1/assignment_loss‚3.273513455282558 /loss/FRt3/detection_loss‚1.4794380961274598 1loss/SRqqt4/detection_loss‚2.6426302077977555 &loss/total_loss‚42.465520244406804ú!MFaçÛ 0loss/FRt3/assignment_loss‚1.1536198216424862 2loss/SRqqt1/assignment_loss‚1.2398282549408979 2loss/SRqqt3/assignment_loss‚1.0173504547238714 0loss/FBt1/assignment_loss‚0.5232646361691877 /loss/FRt2/detection_loss‚2.3044414508033872 /loss/FRt1/assignment_loss‚3.273513455282558 /loss/FRt3/detection_loss‚1.4794380961274598 1loss/SRqqt4/detection_loss‚2.6426302077977555 &loss/total_loss‚42.465520244406804 /loss/FRt2/assignment_loss‚3.418939377399201 1loss/FBt2/assignment_loss‚0.19104851001539203 1loss/FBt3/assignment_loss‚0.05988144868176156 /loss/FBt4/detection_loss‚1.9665325113705225 trainer/global_step‚4099 % _timestamp‚1.7884687585990407e+09 .loss/FRt1/detection_loss‚2.555358113213019 _step‚336 /loss/FRt4/assignment_loss‚2.733195667465528 1loss/FBt4/assignment_loss‚0.18315755160827912 .loss/FRt4/detection_loss‚2.970693210760752 /loss/FBt3/detection_loss‚0.5843703239161062 epoch‚3 2loss/SRqqt2/assignment_loss‚1.3924909176365021 2loss/SRqqt4/assignment_loss‚2.4585830776587776 /loss/FBt1/detection_loss‚2.9172587022185326 .loss/FBt2/detection_loss‚2.021289326250553 _runtime‚4811.783827035 1loss/SRqqt1/detection_loss‚2.0135734067406763 0loss/SRqqt2/detection_loss‚2.357285024658326 1loss/SRqqt3/detection_loss‚1.0077766973252704ПW£Ù‘è‚(:† ™¼çÔ¬ðÛ‡ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:ñS2`é‚(:Õ ™¼çÔ‰ disk.sda2.in‚3.625 disk.md0p1.in‚ 7.12109375 disk.rbd2.in‚0 disk.dm-0.in‚528701.00390625 disk.sda1.in‚0 disk.sdb2.out‚ 522.890625 disk.nvme1n1.in‚ 200395.53125) proc.memory.percent‚18.49517822265625 disk.rbd2.out‚ 1.341796875 disk.sda.in‚3.625 disk.sda.out‚ 522.890625# disk.nvme0n1.in‚328305.47265625 disk.dm-0.out‚61952.71484375 disk.sda2.out‚ 522.890625 network.recv‚ 1.626362e+06) disk./.usagePercent‚83.15614850183763 disk.rbd0.in‚0.015625 disk.sdb.out‚ 522.890625# disk.nvme1n1.out‚40165.41015625 proc.cpu.threads‚17 disk.rbd0.out‚ 1.404296875 disk.sda1.out‚0 disk.md0p1.out‚ 504.3828125$ memory_percent‚34.97929573059082 network.sent‚ 9.53622e+06% disk./.usageGB‚2974.0666580200195 disk.sdb2.in‚ 3.49609375 disk.sdb.in‚ 3.49609375 disk.sdb1.in‚0) proc.memory.availableMB‚ 10652.9921875 proc.memory.rssMB‚3030.25 cpu‚25.15162244119376 disk.md0.in‚ 7.12109375 disk.md0.out‚ 504.3828125 disk.sdb1.out‚0" disk.nvme0n1.out‚ 21787.3046875‡z P’ê‚(:‡ ¨¼çÔÅ劆 gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚137.693' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.42325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|3mlë‚(:á ¨¼çÔæ¤ê† disk.nvme1n1.in‚ 200395.53125 disk.sda.in‚3.625$ memory_percent‚34.97962951660156) proc.memory.percent‚18.49517822265625 disk.sdb1.out‚0 disk.rbd0.out‚ 1.404296875 disk.rbd2.out‚ 1.341796875 disk.sdb1.in‚0# disk.nvme0n1.in‚328305.47265625 disk.rbd0.in‚0.015625 disk.sda2.out‚524.3232421875 disk.md0p1.out‚ 505.7578125 disk.sda1.in‚0 proc.memory.rssMB‚3030.25 network.recv‚ 1.633109e+06 disk.sda1.out‚0 disk.dm-0.out‚ 62085.59375) disk./.usagePercent‚83.15629953315111% disk./.usageGB‚2974.0720596313477 disk.md0p1.in‚ 7.12109375# disk.nvme1n1.out‚40165.60546875 disk.sdb.out‚524.3232421875 disk.sda.out‚524.3232421875 disk.dm-0.in‚528701.00390625 network.sent‚ 9.631031e+06 disk.sda2.in‚3.625 disk.md0.out‚ 505.7578125 disk.rbd2.in‚0 disk.sdb.in‚ 3.49609375& proc.memory.availableMB‚ 10652.9375 cpu‚25.15123476167872 proc.cpu.threads‚17 disk.sdb2.in‚ 3.49609375 disk.sdb2.out‚524.3232421875 disk.md0.in‚ 7.12109375# disk.nvme0n1.out‚21919.98828125ñ{––‘ì‚(:† ·¼çÔ³áÒ‡ gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚135.802' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.9505 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶•¾sí‚(:è ·¼çÔõ¶’‰% disk./.usageGB‚2974.1970558166504 disk.rbd2.out‚ 1.341796875 disk.sdb1.out‚0 disk.md0.out‚ 508.20703125 network.sent‚ 9.636275e+06 disk.rbd0.in‚0.015625 disk.sda2.in‚3.625 disk.rbd0.out‚ 1.404296875 disk.sdb.out‚526.8349609375* proc.memory.availableMB‚10653.18359375 network.recv‚ 1.636686e+06) disk./.usagePercent‚83.15979447880714 disk.rbd2.in‚0 disk.sdb1.in‚0 disk.dm-0.in‚528701.00390625 disk.dm-0.out‚62192.52734375 disk.sda2.out‚526.8349609375 disk.md0p1.out‚ 508.20703125 disk.sdb2.in‚ 3.49609375 disk.sda.in‚3.625 disk.nvme0n1.out‚ 22026.71875 disk.md0.in‚ 7.12109375) proc.memory.percent‚18.49517822265625 cpu‚25.150847385605623 disk.sdb.in‚ 3.49609375# disk.nvme0n1.in‚328305.47265625 disk.sda1.out‚0 disk.sdb2.out‚526.8349609375$ memory_percent‚34.97812747955322 proc.memory.rssMB‚3030.25# disk.nvme1n1.out‚40165.80859375 disk.sda1.in‚0 disk.nvme1n1.in‚ 200395.53125 disk.sda.out‚526.8349609375 disk.md0p1.in‚ 7.12109375 proc.cpu.threads‚176$%8î‚(:„ ƼçÔ´Èé… gpu.0.gpu‚7.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚146.048' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.512 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 K=Qmï‚(:â ƼçÔù÷ï†# disk.nvme1n1.out‚40165.91796875$ memory_percent‚34.98108386993408 disk.sda1.in‚0 disk.sdb1.in‚0 disk.sda.out‚ 528.310546875 disk.md0.out‚ 509.62109375 disk.nvme1n1.in‚ 200395.53125( disk./.usagePercent‚83.1556114660584 disk.dm-0.in‚528701.00390625 disk.sda2.in‚3.625 proc.memory.rssMB‚3030.25$ disk./.usageGB‚2974.047451019287 disk.sda.in‚3.625 disk.sdb2.out‚ 528.310546875 disk.md0.in‚ 7.12109375* proc.memory.availableMB‚10652.69921875 network.sent‚ 9.642592e+06 disk.sda1.out‚0 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 1.404296875 disk.rbd2.in‚0 network.recv‚ 1.641129e+06 disk.sdb1.out‚0! disk.nvme0n1.out‚ 22231.265625 disk.md0p1.out‚ 509.62109375 disk.sdb2.in‚ 3.49609375 disk.sdb.out‚ 528.310546875) proc.memory.percent‚18.49517822265625 cpu‚25.15046036918223# disk.nvme0n1.in‚328305.47265625 disk.sda2.out‚ 528.310546875 disk.rbd2.out‚ 1.341796875 disk.sdb.in‚ 3.49609375 disk.dm-0.out‚62397.18359375 disk.md0p1.in‚ 7.12109375 proc.cpu.threads‚17£.Íq2ð- _wandbruntime‚4863  _runtime‚4863Æ"ZÇñÁ & lr-AdamW/pg1‚0.0006846463154607367 & lr-AdamW/pg2‚0.0006846463154607367 trainer/global_step‚4149 % _timestamp‚1.7884688105403707e+09 _runtime‚4863.725386618 _step‚337aR‘ÌòÆ & lr-AdamW/pg1‚0.0006846463154607367 & lr-AdamW/pg2‚0.0006846463154607367 trainer/global_step‚4149 % _timestamp‚1.7884688105403707e+09 _runtime‚4863.725386618 _step‚337Ñ×YT;2ó- _wandbruntime‚4863  _runtime‚4863[åÆ^à'ôÚO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚168737217757184 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚187126109690368 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78760937 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8217085374652 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47044916 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112727255 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚176977641563136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8240441808896 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76973751 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚176977893234688 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8240675477504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8381854831616 trainer/global_step‚4149 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78760939 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚178743946590720 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚187125801422336 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚168737217757184 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚187126109690368 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78760937 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8217085374652 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47044916 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112727255 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚176977641563136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8240441808896 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76973751 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚176977893234688 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8240675477504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8381854831616 trainer/global_step‚4149 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78760939 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚178743946590720 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚187125801422336 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚168776485791232 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47055718 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76991757 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47055720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8383853436928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚168776485791232 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191533159 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112780317 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112753947 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29936037 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29935357 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8384139848704 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚160632876389312 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8219272844930 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚682 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191559531 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76991075 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚187169794349568 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _step‚339 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚177019143310848 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚178785654500864 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8242405834752 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚187169483997696 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚178785630560768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78779214 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚160632893428672 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191533159 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 % _timestamp‚1.7884688115457053e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78779212 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚177019143310848 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78779214 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78779212 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8219043863638 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191559531 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚4864.732529296 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8242639516672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309ºp!jï&øéM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191533159 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112780317 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112753947 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29936037 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29935357 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8384139848704 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚160632876389312 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8219272844930 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚682 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191559531 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76991075 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚187169794349568 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _step‚339 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚177019143310848 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚178785654500864 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8242405834752 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚187169483997696 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚178785630560768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78779214 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚160632893428672 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191533159 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 % _timestamp‚1.7884688115457053e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78779212 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚177019143310848 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78779214 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78779212 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8219043863638 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191559531 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚4864.732529296 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8242639516672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚168776503794176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8242639516672 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚168776485791232 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47055718 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76991757 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47055720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8383853436928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚168776485791232 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496Óž_R 2ù- _wandbruntime‚4864  _runtime‚4864Ò\ÏÀXúÒ 2loss/SRqqt4/assignment_loss‚1.6488687116652727 1loss/FBt4/assignment_loss‚0.48564663502749755 .loss/FRt2/detection_loss‚2.370500929538722 /loss/FRt4/detection_loss‚2.9133709028915122 0loss/SRqqt4/detection_loss‚2.645237908281129 /loss/FBt1/detection_loss‚2.2969294488430023 epoch‚3 _step‚340 .loss/FBt4/detection_loss‚2.256037838757038 % _timestamp‚1.7884688115499668e+09 /loss/FBt3/detection_loss‚0.6521769859283776 .loss/FRt3/detection_loss‚1.317923673070394 0loss/SRqqt1/detection_loss‚2.159959910846338 1loss/SRqqt2/detection_loss‚2.3776491131101336 0loss/SRqqt3/detection_loss‚0.803981167562306 trainer/global_step‚4149 _runtime‚4864.734974568 /loss/FRt2/assignment_loss‚3.482822687197954 .loss/FRt4/assignment_loss‚3.08287830485238 2loss/SRqqt1/assignment_loss‚1.7259372864372846 1loss/FBt1/assignment_loss‚0.17220103855001817 2loss/FBt3/assignment_loss‚0.013664405727837817 1loss/SRqqt2/assignment_loss‚2.045144435337612 2loss/SRqqt3/assignment_loss‚0.9510299759823829 1loss/FBt2/assignment_loss‚0.23753676711513094 .loss/FRt1/detection_loss‚2.438249740658737 /loss/FBt2/detection_loss‚2.2673476472496987 $loss/total_loss‚42.7740724374443 0loss/FRt1/assignment_loss‚3.1799940815786036 0loss/FRt3/assignment_loss‚1.2489828412349409ŽÉxr]û× .loss/FBt4/detection_loss‚2.256037838757038 % _timestamp‚1.7884688115499668e+09 /loss/FBt3/detection_loss‚0.6521769859283776 .loss/FRt3/detection_loss‚1.317923673070394 0loss/SRqqt1/detection_loss‚2.159959910846338 1loss/SRqqt2/detection_loss‚2.3776491131101336 0loss/SRqqt3/detection_loss‚0.803981167562306 trainer/global_step‚4149 _runtime‚4864.734974568 /loss/FRt2/assignment_loss‚3.482822687197954 .loss/FRt4/assignment_loss‚3.08287830485238 2loss/SRqqt1/assignment_loss‚1.7259372864372846 1loss/FBt1/assignment_loss‚0.17220103855001817 2loss/FBt3/assignment_loss‚0.013664405727837817 1loss/SRqqt2/assignment_loss‚2.045144435337612 2loss/SRqqt3/assignment_loss‚0.9510299759823829 1loss/FBt2/assignment_loss‚0.23753676711513094 .loss/FRt1/detection_loss‚2.438249740658737 /loss/FBt2/detection_loss‚2.2673476472496987 $loss/total_loss‚42.7740724374443 0loss/FRt1/assignment_loss‚3.1799940815786036 0loss/FRt3/assignment_loss‚1.2489828412349409 2loss/SRqqt4/assignment_loss‚1.6488687116652727 1loss/FBt4/assignment_loss‚0.48564663502749755 .loss/FRt2/detection_loss‚2.370500929538722 /loss/FRt4/detection_loss‚2.9133709028915122 0loss/SRqqt4/detection_loss‚2.645237908281129 /loss/FBt1/detection_loss‚2.2969294488430023 epoch‚3 _step‚340Ô ÈX‘ü‚(:† Õ¼çÔ¨ƒþˆ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚135.802' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.9505 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?µzxý‚(:í Õ¼çÔ…˜ÿ‰ disk.sdb.out‚529.81494140625 disk.nvme0n1.out‚ 22407.84375 disk.sdb2.in‚ 3.49609375% memory_percent‚34.979796409606934 disk.rbd2.in‚0 disk.rbd2.out‚ 1.341796875 disk.sdb.in‚ 3.49609375 disk.sdb1.in‚0 disk.sda1.out‚0! disk.sda2.out‚529.81494140625 disk.md0.out‚ 511.05078125) proc.memory.percent‚18.49517822265625# disk.nvme1n1.out‚40166.16796875 disk.sdb1.out‚0 disk.sda.out‚529.81494140625# disk.nvme0n1.in‚328305.47265625 disk.md0.in‚ 7.12109375 proc.cpu.threads‚17 disk.rbd0.out‚ 1.404296875 disk.md0p1.out‚ 511.05078125 cpu‚25.150069012432933 disk.rbd0.in‚0.015625 disk.sda.in‚3.625 disk.sda1.in‚0 network.recv‚ 1.645247e+06 proc.memory.rssMB‚3030.25 network.sent‚ 9.661242e+06) disk./.usagePercent‚83.16134126984397 disk.dm-0.in‚528701.00390625 disk.md0p1.in‚ 7.12109375% disk./.usageGB‚2974.2523765563965 disk.nvme1n1.in‚ 200395.53125 disk.dm-0.out‚62574.01171875 disk.sda2.in‚3.625! disk.sdb2.out‚529.81494140625* proc.memory.availableMB‚10652.91015625 Fe€þ‚(:… ä¼çÔµæÙˆ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚121.536' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.384 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M/ß rÿ‚(:ç ä¼çÔ鬆‹ disk.sda1.in‚0 disk.rbd0.out‚ 1.404296875 disk.sda1.out‚0 disk.md0p1.out‚ 512.97265625 disk.md0.out‚ 512.97265625 disk.rbd2.in‚0 proc.memory.rssMB‚3030.25 cpu‚25.149701954708878 disk.sda.out‚531.8076171875# disk.nvme0n1.in‚328305.47265625 disk.sdb2.in‚ 3.49609375) proc.memory.percent‚18.49517822265625 disk.sda2.in‚3.625 disk.md0.in‚ 7.12109375 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 1.341796875 disk.sdb.in‚ 3.49609375% memory_percent‚34.979844093322754 network.recv‚ 1.652203e+06 disk.sda.in‚3.625 disk.md0p1.in‚ 7.12109375 disk.nvme1n1.in‚ 200395.53125 disk.dm-0.in‚528701.00390625 disk.dm-0.out‚ 62775.671875 disk.sdb1.in‚0 disk.sdb1.out‚0* proc.memory.availableMB‚10652.90234375 network.sent‚ 9.758154e+06' disk./.usagePercent‚83.163270545549$ disk./.usageGB‚2974.321376800537# disk.nvme0n1.out‚22609.37890625 disk.sda2.out‚531.8076171875 disk.sdb2.out‚531.8076171875# disk.nvme1n1.out‚40166.29296875 disk.sdb.out‚531.8076171875 proc.cpu.threads‚17Çú@’€‚(:‡ ó¼çÔç¡£‡ gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚144.951' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.23775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Fµx‚(:í ó¼çÔ÷¶Àˆ network.recv‚ 1.656099e+06% disk./.usageGB‚2974.9461517333984 disk.md0.in‚ 7.12109375 disk.rbd2.in‚0# disk.nvme0n1.in‚328305.47265625! disk.sda2.out‚533.25048828125 disk.md0p1.in‚ 7.12109375 disk.rbd0.in‚0.015625# disk.nvme1n1.out‚40166.72265625 disk.sdb.out‚533.25048828125 disk.sdb1.out‚0% memory_percent‚34.979796409606934 cpu‚25.14932521337473) disk./.usagePercent‚83.18073951416042 disk.rbd2.out‚ 1.341796875 disk.sda.out‚533.25048828125 disk.sda1.in‚0 disk.md0p1.out‚ 514.35546875 disk.nvme1n1.in‚ 200395.53125 disk.sdb1.in‚0 disk.sda1.out‚0! disk.sdb2.out‚533.25048828125 disk.sdb.in‚ 3.49609375 disk.sda.in‚3.625 disk.dm-0.out‚62987.86328125 disk.sda2.in‚3.625* proc.memory.availableMB‚10652.91015625) proc.memory.percent‚18.49517822265625 disk.sdb2.in‚ 3.49609375 disk.md0.out‚ 514.35546875! disk.nvme0n1.out‚ 22821.140625 disk.dm-0.in‚528701.00390625 proc.cpu.threads‚17 network.sent‚ 9.763481e+06 disk.rbd0.out‚ 1.404296875 proc.memory.rssMB‚3030.25¡Ôìs2‚- _wandbruntime‚4919  _runtime‚4919·Í%@ǃÁ & lr-AdamW/pg1‚0.0006801517864959249 & lr-AdamW/pg2‚0.0006801517864959249 trainer/global_step‚4199 % _timestamp‚1.7884688658844216e+09 _runtime‚4919.069522262 _step‚341ÒÄH}Ì„Æ trainer/global_step‚4199 % _timestamp‚1.7884688658844216e+09 _runtime‚4919.069522262 _step‚341 & lr-AdamW/pg1‚0.0006801517864959249 & lr-AdamW/pg2‚0.0006801517864959249ÕZ†i2…- _wandbruntime‚4919  _runtime‚4919þd²†ÚO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170701519606784 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚162471880942448 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8338891184128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79674687 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8338657515520 _runtime‚4919.070876429 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8338891184128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77861721 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47585016 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179040159119360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114065207 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47585018 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚170701519606784 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚170786887318248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segmDÊÝá.ent.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193739894 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77862395 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79674689 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚180828169033216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 193766240 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚189309829271040 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _step‚342©‹«å'‡ßO trainer/global_step‚4199 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚179040410790912 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30276705 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8315023415160 % _timestamp‚1.7884688658855574e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79674689 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8481968505856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚179040410790912 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8481684177920 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8315252396244 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79674687 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8338657515520 _runtime‚4919.070876429 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8338891184128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77861721 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47585016 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179040159119360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114065207 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47585018 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚170701519606784 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚170786887318248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193739894 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77862395 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79674689 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚180828169033216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 193766240 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚189309829271040 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _step‚342 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170701519606784 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚162471880942448 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8338891184128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193811233 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77879910 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚189353823511040 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 % _timestamp‚1.7884688669691703e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚179081409182208 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚170740787640832 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79692964 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47595818 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30283399 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8483673610240 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8316981904146 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚170740805643776 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 trainer/global_step‚4199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _runtime‚4920.155365304 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8340855223296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚170826376164658 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79692964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114118269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚691 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8483970507776 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚162509394260512 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30284090 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193784861 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77879217 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚170740805643776 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8340855223296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚170740787640832 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114118269 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚180869853003264 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚162509411299872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚179081660867072 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚180869829063168 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 _step‚343 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193811233 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77879910 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚189353823511040 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 % _timestamp‚1.7884688669691703e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚179081409182208 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚170740787640832 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79692964 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47595818 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30283399 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8483673610240 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8316981904146 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚170740805643776 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 trainer/global_step‚4199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _runtime‚4920.155365304 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8340855223296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚170826376164658 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79692964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114118269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚691 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8483970507776 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚162509394260512 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30284090 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193784861 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77879217 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚170740805643776 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8340855223296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚170740787640832 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114118269 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚180869853003264 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚162509411299872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚179081660867072 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚180869829063168 PDeviceStatsMonitor.on_J‰´ÑRtrain_batch_end/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 _step‚343 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚172761589250400 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚164348398813648 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚181102928347136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8413190436752 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _step‚346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80588439 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78747766 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚191494262450688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48125118 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 $ _timestamp‚1.788468921328441e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚172665803453440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78747093 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30622648 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚191493945794048 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8412961455668 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8436873222144 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚172665821456384 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚181102928347136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181102676675584 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115403159 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚182912343595520 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚172665821456384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8437106890752 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115429503 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300|Zøä'›ÞO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚191494262450688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48125118 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 $ _timestamp‚1.788468921328441e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚172665803453440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78747093 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30622648 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚191493945794048 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8412961455668 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8436873222144 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚172665821456384 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚181102928347136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181102676675584 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115403159 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚182912343595520 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚172665821456384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8437106890752 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115429503 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚164348381774288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚671 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _runtime‚4974.513676193 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 195991596 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80588437 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚181102676675584 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚673 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚172761343229956 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 trainer/global_step‚4249 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8436873222144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8581894915072 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚172761589250400 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚164348398813648 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚181102928347136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8413190436752 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _step‚346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80588439 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78747766 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444ÚªXa¥2œ- _wandbruntime‚4975  _runtime‚49759÷9^ äM JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115456221 trainer/global_step‚4249 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196062935 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚172705107493376 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚172705107493376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48135918 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚182954051505664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80606712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80606714 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30628520 _HŒKDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚182954027565568 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8438837248000 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78764438 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8583884678144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8583596169216 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚172801078097018 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚181143926738432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8438837248000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80606712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115429851 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115456221 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8414919944654 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚181144178423296 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚172705089490432 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8439070929920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚664 _step‚347 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196036563 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30629184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚164385912131712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8415148925946 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0¸îSÉï&žéM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78764438 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8583884678144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8583596169216 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚172801078097018 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚181143926738432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8438837248000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80606712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115429851 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115456221 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8414919944654 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚181144178423296 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚172705089490432 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8439070929920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚664 _step‚347 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196036563 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30629184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚164385912131712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8415148925946 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115456221 trainer/global_step‚4249 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196062935 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚172705107493376 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚172705107493376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48135918 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚182954051505664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80606712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80606714 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30628520 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚182954027565568 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8438837248000 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944ÛÞ/øï2Ÿ- _wandbruntime‚4975  _runtime‚4975yÑm^Y Ó .loss/FBt1/detection_loss‚3.365403786301613 &loss/total_loss‚46.173461409367306 trainer/global_step‚4249 2loss/SRqqt3/assignment_loss‚1.2945758978215356 0loss/FBt1/assignment_loss‚0.4916277192533016 /loss/FRt2/detection_loss‚2.2250519283976327 .loss/FBt4/detection_loss‚2.865783068266782 1loss/SRqqt4/detection_loss‚2.5782275391476497 _step‚348 2loss/SRqqt2/assignment_loss‚1.5876620188355446 1loss/FBt4/assignment_loss‚0.06636774318063875 -loss/FBt2/detection_loss‚2.40227984637022 % _timestamp‚1.7884689223632636e+09 /loss/FRt4/assignment_loss‚4.503806858062744 _runtime‚4975.547961986 1loss/SRqqt1/detection_loss‚1.7800262732939287 /loss/FBt3/detection_loss‚0.7380963315797114 epoch‚3 0loss/FRt3/assignment_loss‚1.2831680566975565 2loss/SRqqt1/assignment_loss‚1.6222782656550407 2loss/SRqqt4/assignment_loss‚2.3789107772920812 /loss/FRt1/detection_loss‚3.0846840102097084 .loss/FRt3/detection_loss‚1.421545875230522 -loss/FRt4/detection_loss‚3.21317360162735 0loss/SRqqt2/detection_loss‚2.189607377235706 0loss/FRt1/assignment_loss‚2.7364865048178313 /loss/FRt2/assignment_loss‚3.380195271709691 1loss/FBt2/assignment_loss‚0.09830631114652988 1loss/FBt3/assignment_loss‚0.13510914409641175 1loss/SRqqt3/detection_loss‚0.7310872031375766âbË^¡Ø epoch‚3 0loss/FRt3/assignment_loss‚1.2831680566975565 2loss/SRqqt1/assignment_loss‚1.6222782656550407 2loss/SRqqt4/assignment_loss‚2.3789107772920812 /loss/FRt1/detection_loss‚3.0846840102097084 .loss/FRt3/detection_loss‚1.421545875230522 -loss/FRt4/detection_loss‚3.21317360162735 0loss/SRqqt2/detection_loss‚2.189607377235706 0loss/FRt1/assignment_loss‚2.7364865048178313 /loss/FRt2/assignment_loss‚3.380195271709691 1loss/FBt2/assignment_loss‚0.09830631114652988 1loss/FBt3/assignment_loss‚0.13510914409641175 1loss/SRqqt3/detection_loss‚0.7310872031375766 .loss/FBt1/detection_loss‚3.365403786301613 &loss/total_loss‚46.173461409367306 trainer/global_step‚4249 2loss/SRqqt3/assignment_loss‚1.2945758978215356 0loss/FBt1/assignment_loss‚0.4916277192533016 /loss/FRt2/detection_loss‚2.2250519283976327 .loss/FBt4/detection_loss‚2.865783068266782 1loss/SRqqt4/detection_loss‚2.5782275391476497 _step‚348 2loss/SRqqt2/assignment_loss‚1.5876620188355446 1loss/FBt4/assignment_loss‚0.06636774318063875 -loss/FBt2/detection_loss‚2.40227984637022 % _timestamp‚1.7884689223632636e+09 /loss/FRt4/assignment_loss‚4.503806858062744 _runtime‚4975.547961986 1loss/SRqqt1/detection_loss‚1.7800262732939287 /loss/FBt3/detection_loss‚0.7380963315797114Üvœ¹ë’¢‚(:‡ ¾½çԛݗ‡ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚140.145' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.03625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ6¥äl£‚(:á ¾½çÔÏ‘à‡' disk./.usagePercent‚83.176580820013 disk.rbd2.in‚0 disk.sdb.in‚ 3.49609375# disk.nvme0n1.in‚328305.47265625 disk.dm-0.in‚528701.00390625 disk.sdb2.out‚ 541.599609375 disk.md0.in‚ 7.12109375 disk.rbd0.out‚ 1.404296875 disk.sdb1.in‚0# memory_percent‚34.9851131439209 proc.cpu.threads‚17 disk.sda.in‚3.625 disk.sda1.out‚0 disk.sdb1.out‚0) proc.memory.availableMB‚ 10652.0390625 network.recv‚ 1.682559e+06 disk.md0.out‚ 522.40234375 disk.rbd2.out‚ 1.341796875 disk.nvme1n1.in‚ 200395.53125 cpu‚25.147597851089554 network.sent‚ 9.972267e+06 disk.sda.out‚ 541.599609375 disk.dm-0.out‚64185.81640625# disk.nvme0n1.out‚24018.11328125 disk.sda2.out‚ 541.599609375 disk.sdb2.in‚ 3.49609375! disk.nvme1n1.out‚ 40167.703125 disk.sda1.in‚0 disk.md0p1.in‚ 7.12109375 disk.md0p1.out‚ 522.40234375 disk.rbd0.in‚0.015625 proc.memory.rssMB‚3030.25% disk./.usageGB‚2974.7974166870117 disk.sda2.in‚3.625 disk.sdb.out‚ 541.599609375) proc.memory.percent‚18.49517822265625æ‹Y¢‘¤‚(:† ͽçÔЈ‡ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚138.642' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.6605 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ï-œ-t¥‚(:é ͽçÔÂËÔ‡ disk.sdb2.in‚ 3.49609375! disk.sdb2.out‚543.37646484375$ disk./.usageGB‚2973.585792541504! disk.sda2.out‚543.37646484375 disk.md0p1.in‚ 7.12109375 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.dm-0.in‚528701.00390625 disk.dm-0.out‚64314.66796875 disk.md0p1.out‚ 524.12109375 disk.sdb.out‚543.37646484375 disk.sda.in‚3.625* proc.memory.availableMB‚10652.54296875) proc.memory.percent‚18.49517822265625 network.sent‚ 1.0080801e+07 disk.rbd0.in‚0.015625 disk.nvme1n1.in‚ 200395.53125 disk.sdb.in‚ 3.49609375 disk.nvme0n1.out‚24146.25 proc.memory.rssMB‚3030.25 disk.sda1.in‚0 disk.md0.out‚ 524.12109375 disk.rbd0.out‚ 1.404296875 disk.rbd2.out‚ 1.341796875 network.recv‚ 1.688113e+06) disk./.usagePercent‚83.14270330180051 disk.sda1.out‚0 disk.sda2.in‚3.625 disk.rbd2.in‚0 cpu‚25.147208116997362 disk.md0.in‚ 7.12109375# disk.nvme1n1.out‚40168.41796875 disk.sda.out‚543.37646484375# disk.nvme0n1.in‚328305.47265625$ memory_percent‚34.98203754425049 proc.cpu.threads‚17øf!’¦‚(:‡ ܽçÔíÊé… gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚138.919' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.72975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³È“+p§‚(:å ܽçÔåœô† disk.nvme1n1.in‚ 200395.53125 disk.sdb.out‚544.88330078125# disk.nvme0n1.in‚328305.47265625! disk.sdb2.out‚544.88330078125 proc.memory.rssMB‚3030.25 cpu‚25.146827733785074 proc.cpu.threads‚17 network.recv‚ 1.691984e+06 disk.dm-0.in‚528701.00390625 disk.dm-0.out‚64495.32421875( proc.memory.availableMB‚ 10652.546875) proc.memory.percent‚18.49517822265625 disk.rbd2.in‚0 disk.sda1.out‚0 disk.md0p1.out‚ 525.5703125 disk.sdb2.in‚ 3.49609375 disk.md0.in‚ 7.12109375 disk.sdb.in‚ 3.49609375 disk.sda.in‚3.625 disk.md0p1.in‚ 7.12109375 disk.rbd0.in‚0.015625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0.out‚ 525.5703125 network.sent‚ 1.0084534e+07$ disk./.usageGB‚2973.764938354492 disk.rbd2.out‚ 1.341796875 disk.sda.out‚544.88330078125 disk.rbd0.out‚ 1.404296875# disk.nvme1n1.out‚40168.54296875 disk.sda2.in‚3.625! disk.sda2.out‚544.88330078125& disk./.usagePercent‚83.14771229371 disk.nvme0n1.out‚ 24326.78125 disk.sda1.in‚0$ memory_percent‚34.98201370239258µÈ  ‘¨‚(:† ë½çÔÔ§‡ gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚136.134' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.0335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02Œ)ws©‚(:è ë½çÔãƒÜ‡ disk.sda2.in‚3.625) proc.memory.availableMB‚ 10652.9140625 proc.cpu.threads‚17$ disk./.usageGB‚2973.890239715576 disk.md0.in‚ 7.12109375 disk.rbd0.in‚0.015625 disk.nvme1n1.in‚ 200395.53125# disk.nvme0n1.in‚328305.47265625 disk.sda1.in‚0! disk.sdb2.out‚546.40283203125 cpu‚25.146433314018243 disk.rbd0.out‚ 1.404296875 disk.sda.in‚3.625 disk.sda.out‚546.40283203125 disk.dm-0.in‚528701.00390625 disk.dm-0.out‚ 64548.71875 disk.md0p1.in‚ 7.12109375( disk./.usagePercent‚83.1512157722086 disk.md0p1.out‚ 527.04296875 disk.sdb2.in‚ 3.49609375" disk.nvme1n1.out‚ 40168.7734375! disk.sda2.out‚546.40283203125 proc.memory.rssMB‚3030.25) proc.memory.percent‚18.49517822265625 network.sent‚ 1.0089861e+07 disk.md0.out‚ 527.04296875 network.recv‚ 1.695986e+06 disk.sdb1.out‚0$ memory_percent‚34.97977256774902 disk.rbd2.out‚ 1.341796875 disk.sdb.out‚546.40283203125 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.sdb.in‚ 3.49609375" disk.nvme0n1.out‚ 24379.9453125 disk.sda1.out‚0jOc2ª- _wandbruntime‚5026  _runtime‚5026·{hÇ«Á & lr-AdamW/pg2‚0.0006709961710778864 trainer/global_step‚4299 % _timestamp‚1.7884689732885087e+09 _runtime‚5026.473560674 _step‚349 & lr-AdamW/pg1‚0.0006709961710778864’nX¦Ì¬Æ % _timestamp‚1.7884689732885087e+09 _runtime‚5026.473560674 _step‚349 & lr-AdamW/pg1‚0.0006709961710778864 & lr-AdamW/pg2‚0.0006709961710778864 trainer/global_step‚4299Ý‘Ž 2­- _wandbruntime‚5026  _runtime‚5026†(K"®ÚO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79634306 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116767455 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30969088 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8681716591104 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚166224899645488 _runtime‚5026.474929061 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚174630123305984 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8535088928768 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚682 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚184996566038016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚183165194231808 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚174630105303040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48665218 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48665216 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚183165445903360 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116741111 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚684 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚183165445903360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30968406 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 296910848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198269644 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8510899496176 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚183165194231808 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79633622 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81502187 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚174735799141664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.cu¶i Ârrent‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚174630105303040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8535322597376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚193677961778176 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _step‚350 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 % _timestamp‚1.7884689732897685e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81502189 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 198243298 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚193678282629120 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116767455 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚174736045162108 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8535322597376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8535088928768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81502187 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚174630123305984 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚4299ÄÚ…wå'¯ßO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚184996566038016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚183165194231808 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚174630105303040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48665218 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48665216 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚183165445903360 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116741111 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚684 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚183165445903360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30968406 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 296910848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198269644 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8510899496176 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚183165194231808 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79633622 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81502187 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚174735799141664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚174630105303040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8535322597376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚193677961778176 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _step‚350 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 % _timestamp‚1.7884689732897685e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81502189 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 198243298 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚193678282629120 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116767455 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚174736045162108 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8535322597376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8535088928768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81502187 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚174630123305984 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚4299 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79634306 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116767455 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30969088 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8681716591104 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚166224899645488 _runtime‚5026.474929061 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚174630123305984 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8535088928768 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚682Þ¾­šè2°- _wandbruntime‚5027  _runtime‚5027=|+þê&±äM aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8683702593024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8537052954624 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198314637 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116794173 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚193721952601088 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48676020 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚351 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚185038250008064 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚174775534008726 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884689742580092e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198288265 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81520462 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8513086966454 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81520464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30975009 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79651027 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48676018 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚183206695979520 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚174669409342976 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116767803 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚673 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚193721635957760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116794173 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚183206444294656 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚671 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116767803 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚174669391340032QX¯"²éM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚183206695979520 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚174669409342976 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116767803 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚673 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚193721635957760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116794173 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚183206444294656 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚671 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116767803 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚174669391340032 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8683702593024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8537052954624 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198314637 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116794173 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚193721952601088 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48676020 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚351 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚185038250008064 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚174775534008726 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884689742580092e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198288265 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81520462 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8513086966454 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81520464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30975009 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79651027 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48676018 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920ßÊÚ¢2³- _wandbruntime‚5027  _runtime‚5027uýU´Ï .loss/FBt1/detection_loss‚2.551879349350929 /loss/FRt2/detection_loss‚2.2197439574007323 .loss/FRt4/assignment_loss‚4.23321420655531 2loss/SRqqt1/assignment_loss‚1.5380106965700786 0loss/SRqqt2/assignment_loss‚1.77262111221041 0loss/FBt4/assignment_loss‚0.8263733027100292 1loss/SRqqt4/detection_loss‚2.7480864993163516 _runtime‚5027.445670414 %loss/total_loss‚45.18657954833006 trainer/global_step‚4299 /loss/FRt1/assignment_loss‚3.206673409257616 2loss/SRqqt4/assignment_loss‚1.6069483985858304 0loss/FBt1/assignment_loss‚0.2377848845670087 1loss/FBt2/assignment_loss‚0.15427964771736566 1loss/SRqqt3/detection_loss‚0.7289380337629053 /loss/FBt4/detection_loss‚3.5022888282934823 epoch‚3 # _timestamp‚1.78846897426096e+09 0loss/FRt2/assignment_loss‚3.3076817719709304 1loss/FBt3/assignment_loss‚0.08690682436557667 .loss/FBt3/detection_loss‚0.717271038536708 .loss/FRt1/detection_loss‚2.830732530781201 1loss/SRqqt2/detection_loss‚2.3500355077641353 0loss/FRt3/assignment_loss‚0.9920699512347196 2loss/SRqqt3/assignment_loss‚0.9218109869449916 .loss/FRt3/detection_loss‚1.440665842535404 .loss/FBt2/detection_loss‚1.600935039775712 /loss/FRt4/detection_loss‚3.6340393704526566 1loss/SRqqt1/detection_loss‚1.9775883576699667 _step‚352[DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚176594425155584 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82415937 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49205316 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚176710255053372 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚176594407152640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31312126 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31312810 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚187080764540416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49205318 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 $ _timestamp‚1.788469027115984e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚176594425155584 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚185227711788032 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82415939 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚185227963459584 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8633538304000 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8608837536684 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185227711788032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚354 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚4349 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚176710501073816 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 200521346 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82415937 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8781290793472 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8633304635392 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118079063 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚185227963459584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82415939 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚185227963459584 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8633538304000 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8608837536684 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185227711788032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚354 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚4349 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚176710501073816 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 200521346 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82415937 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8781290793472 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8633304635392 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118079063 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚185227963459584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚176594425155584 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82415937 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49205316 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚176710255053372 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚176594407152640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31312126 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2âI42Â- _wandbruntime‚5081  _runtime‚5081;Èš´Ø ÃäM ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80535578 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80534892 EDeviV}Ý#ceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118132125 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚195906018637824 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49216120 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 _step‚355 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200566339 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8783570127360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚185269213535744 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 200539967 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82434212 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118132125 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8610796025670 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31318774 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8611025006962 _runtime‚5081.277560112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 trainer/global_step‚4349 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82434214 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚185269213535744 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82434212 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8783287909888 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82434214 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118105755 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 200539967 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273ø¦/˜ï&ÄéM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82434214 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118105755 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 200539967 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80535578 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80534892 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118132125 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚195906018637824 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49216120 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 _step‚355 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200566339 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8783570127360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚185269213535744 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 200539967 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82434212 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118132125 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8610796025670 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31318774 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8611025006962 _runtime‚5081.277560112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 trainer/global_step‚4349 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82434214 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚185269213535744 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82434212 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8783287909888 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0ã²õu2Å- _wandbruntime‚5081  _runtime‚5081Ï2´¸]Æ× % _timestamp‚1.7884690280944037e+09 2loss/SRqqt1/assignment_loss‚1.7547296570970656 1loss/SRqqt3/detection_loss‚0.8868982521696137 _step‚356 /loss/FRt3/detection_loss‚1.5356740207662658 1loss/SRqqt1/detection_loss‚1.9334935178147985 1loss/SRqqt4/assignment_loss‚1.782072803299678 1loss/FBt1/assignment_loss‚0.23191783039198657 %loss/total_loss‚43.30754551863264 epoch‚3 0loss/FRt1/assignment_loss‚3.0868200576967664 0loss/FRt2/assignment_loss‚3.2720426268047755 2loss/SRqqt3/assignment_loss‚1.1738638918750084 .loss/FRt4/detection_loss‚3.097734386170352 1loss/SRqqt4/detection_loss‚2.0481775708888708 2loss/FBt3/assignment_loss‚0.014634983973234934 .loss/FRt2/detection_loss‚2.301033734017336 0loss/SRqqt2/detection_loss‚2.174786791989678 .loss/FBt1/detection_loss‚2.277176124708993 /loss/FBt2/detection_loss‚1.8784516751766205 trainer/global_step‚4349 0loss/FRt3/assignment_loss‚1.3642167240854293 0loss/FRt4/assignment_loss‚3.9792794982592263 1loss/SRqqt2/assignment_loss‚1.930052232977591 _runtime‚5081.279113779 1loss/FBt2/assignment_loss‚0.18608112143262664 1loss/FBt4/assignment_loss‚0.24320431891828775 /loss/FRt1/detection_loss‚2.6544374740786023 /loss/FBt3/detection_loss‚0.7557920082144334 .loss/FBt4/detection_loss‚2.744974215825399áþƒÉbÇÜ 2loss/SRqqt1/assignment_loss‚1.7547296570970656 1loss/SRqqt3/detection_loss‚0.8868982521696137 _step‚356 /loss/FRt3/detection_loss‚1.5356740207662658 1loss/SRqqt1/detection_loss‚1.9334935178147985 1loss/SRqqt4/assignment_loss‚1.782072803299678 1loss/FBt1/assignment_loss‚0.23191783039198657 %loss/total_loss‚43.30754551863264 epoch‚3 0loss/FRt1/assignment_loss‚3.0868200576967664 0loss/FRt2/assignment_loss‚3.2720426268047755 2loss/SRqqt3/assignment_loss‚1.1738638918750084 .loss/FRt4/detection_loss‚3.097734386170352 1loss/SRqqt4/detection_loss‚2.0481775708888708 2loss/FBt3/assignment_loss‚0.014634983973234934 .loss/FRt2/detection_loss‚2.301033734017336 0loss/SRqqt2/detection_loss‚2.174786791989678 .loss/FBt1/detection_loss‚2.277176124708993 /loss/FBt2/detection_loss‚1.8784516751766205 trainer/global_step‚4349 0loss/FRt3/assignment_loss‚1.3642167240854293 0loss/FRt4/assignment_loss‚3.9792794982592263 1loss/SRqqt2/assignment_loss‚1.930052232977591 _runtime‚5081.279113779 1loss/FBt2/assignment_loss‚0.18608112143262664 1loss/FBt4/assignment_loss‚0.24320431891828775 /loss/FRt1/detection_loss‚2.6544374740786023 /loss/FBt3/detection_loss‚0.7557920082144334 .loss/FBt4/detection_loss‚2.744974215825399 % _timestamp‚1.7884690280944037e+09䵌¯ˆ‘È‚(:† §¾çÔôçÕ† gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚129.282' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.3205 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ×Ü4wÉ‚(:ì §¾çÔ¯Œ‚‡ disk.sda2.in‚ 3.62890625) disk./.usagePercent‚83.14847288996455 disk.rbd2.in‚0 disk.rbd2.out‚ 1.341796875 disk.sdb.in‚ 3.49609375 disk.md0.in‚7.125# disk.nvme1n1.in‚200395.54296875# disk.nvme0n1.in‚328311.12109375 disk.dm-0.in‚528706.6640625 disk.md0p1.out‚ 533.5078125 proc.cpu.threads‚17# disk.nvme1n1.out‚40169.72265625 disk.sdb1.in‚0 disk.sda.out‚553.12939453125! disk.sda2.out‚553.12939453125 disk.md0.out‚ 533.5078125 disk.sda.in‚ 3.62890625! disk.sdb2.out‚553.12939453125 proc.memory.rssMB‚3030.25 network.recv‚ 1.715146e+06% disk./.usageGB‚2973.7921409606934 disk.sdb.out‚553.12939453125# disk.nvme0n1.out‚24850.94140625 disk.dm-0.out‚ 65020.6640625 disk.sda1.in‚0) proc.memory.availableMB‚ 10652.2109375) proc.memory.percent‚18.49517822265625 network.sent‚ 1.021479e+07 disk.sdb1.out‚0 disk.sda1.out‚0 disk.sdb2.in‚ 3.49609375 disk.rbd0.out‚ 1.404296875$ memory_percent‚34.98406410217285 disk.md0p1.in‚7.125 disk.rbd0.in‚0.015625 cpu‚25.14482167048719Þ#¤yÊ‚(:î ¶¾çÔ‰£é†% disk./.usageGB‚2973.9125938415527! disk.sdb2.out‚554.61474609375 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.dm-0.in‚528706.6640625 disk.sda2.in‚ 3.62890625 network.sent‚ 1.0322551e+07 disk.sdb2.in‚ 3.49609375 disk.md0.out‚ 534.93359375 disk.md0p1.out‚ 534.93359375$ memory_percent‚34.98415946960449 cpu‚25.14449786176987 disk.rbd2.in‚0 disk.rbd2.out‚ 1.341796875 disk.sdb.in‚ 3.49609375! disk.sda2.out‚554.61474609375) proc.memory.percent‚18.49517822265625 network.recv‚ 1.721846e+06# disk.nvme1n1.out‚40169.91796875 disk.sdb.out‚554.61474609375 disk.sda.out‚554.61474609375# disk.nvme0n1.in‚328311.12109375) disk./.usagePercent‚83.15184080292684 disk.md0.in‚7.125# disk.nvme1n1.in‚200395.54296875 disk.sda.in‚ 3.62890625 disk.dm-0.out‚65027.05859375 disk.md0p1.in‚7.125) proc.memory.availableMB‚ 10652.1953125 proc.memory.rssMB‚3030.25 disk.sda1.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 1.404296875! disk.nvme0n1.out‚ 24857.140625 disk.sda1.out‚0œÌùŠ’Ë‚(:‡ ¶¾çÔ÷”ä† gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚140.145' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.03625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ܳ=’Ì‚(:‡ žçÔìÑŸ† gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚137.693' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.42325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹4…À{Í‚(:ð žçÔÖí£‡ disk.md0.out‚ 536.69921875 disk.sda2.in‚ 3.62890625 proc.memory.rssMB‚3030.25 cpu‚25.144169369955524 disk.rbd2.out‚ 1.341796875) disk./.usagePercent‚83.15628929374003 disk.md0.in‚7.125 disk.sda1.in‚0) proc.memory.percent‚18.49517822265625% memory_percent‚34.985899925231934* proc.memory.availableMB‚10651.91015625 proc.cpu.threads‚17# disk.nvme0n1.in‚328311.12109375 disk.dm-0.out‚65231.23828125 disk.md0p1.out‚ 536.69921875 disk.sdb1.in‚0 disk.sda.out‚556.45166015625! disk.sda2.out‚556.45166015625 disk.sda1.out‚0! disk.sdb2.out‚556.45166015625 disk.rbd0.out‚ 1.404296875# disk.nvme1n1.in‚200395.54296875" disk.nvme1n1.out‚ 40240.8515625 disk.sdb.in‚ 3.49609375 disk.sdb.out‚556.45166015625 disk.sdb1.out‚0 network.recv‚ 1.726259e+06 disk.md0p1.in‚7.125# disk.nvme0n1.out‚24990.38671875 disk.dm-0.in‚528706.6640625 disk.rbd2.in‚0 disk.rbd0.in‚0.015625 network.sent‚ 1.0328361e+07# disk./.usageGB‚2974.07169342041 disk.sdb2.in‚ 3.49609375 disk.sda.in‚ 3.62890625ª½ì’΂(:‡ Ô¾çÔ’…Š† gpu.0.gpu‚42.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚128.665' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.16625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉÆžyÏ‚(:î Ô¾çÔøÿŠ‡ disk.sda.out‚557.78173828125 disk.md0p1.in‚7.125# disk.nvme0n1.out‚25134.44921875 network.sent‚ 1.0334373e+07 disk.md0p1.out‚ 537.9765625# disk.nvme1n1.in‚200395.54296875 disk.sdb1.in‚0) disk./.usagePercent‚83.16677221081855 disk.md0.out‚ 537.9765625 disk.rbd0.out‚ 1.404296875! disk.sdb2.out‚557.78173828125 disk.sdb.out‚557.78173828125 disk.sdb1.out‚0% memory_percent‚34.984421730041504* proc.memory.availableMB‚10652.15234375 disk.sda2.in‚ 3.62890625 network.recv‚ 1.730603e+06% disk./.usageGB‚2974.4466133117676 disk.dm-0.in‚528706.6640625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0 proc.cpu.threads‚17# disk.nvme0n1.in‚328311.12109375 disk.dm-0.out‚ 65502.4609375 disk.sdb2.in‚ 3.49609375 disk.md0.in‚7.125 disk.rbd0.in‚0.015625 proc.memory.rssMB‚3030.25 cpu‚25.1438008849687! disk.sda2.out‚557.78173828125 disk.sda.in‚ 3.62890625 disk.rbd2.out‚ 1.341796875# disk.nvme1n1.out‚40368.01171875 disk.sdb.in‚ 3.49609375) proc.memory.percent‚18.49517822265625úì“ð2Ð- _wandbruntime‚5131  _runtime‚5131”¨ÔÇÑÁ _runtime‚5131.856107935 _step‚357 & lr-AdamW/pg1‚0.0006616248579276613 & lr-AdamW/pg2‚0.0006616248579276613 trainer/global_step‚4399 % _timestamp‚1.7884690786712122e+09ø06óÌÒÆ & lr-AdamW/pg2‚0.0006616248579276613 trainer/global_step‚4399 % _timestamp‚1.7884690786712122e+09 _runtime‚5131.856107935 _step‚357 & lr-AdamW/pg1‚0.0006616248579276613厛 º2Ó- _wandbruntime‚5131  _runtime‚5131m¾¸AnÔÙO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚187290481015808 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8881406599680 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚187290229344256 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚169977935387888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 $ _timestamp‚1.788469078671938e+09 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81408079 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119443359 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚178558727005184 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚178684956985524 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚169977952427248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚198046059277312 =7DeviceStatsMonitor.on_train_batch_start/num_device_fz-” qree‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚178684710965080 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83329687 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83329689 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49745418 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8881120174592 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49745416 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚187290481015808 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8731754010624 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202773048 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8731520342016 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚187290229344256 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚4399 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624¢Vä'ÕÞO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83329689 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49745418 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8881120174592 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49745416 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚187290481015808 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8731754010624 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202773048 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8731520342016 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚187290229344256 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚4399 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚187290481015808 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8881406599680 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚187290229344256 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚169977935387888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 $ _timestamp‚1.788469078671938e+09 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81408079 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119443359 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚178558727005184 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚178684956985524 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚169977952427248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚198046059277312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚178684710965080 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83329687 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 _runtime‚ 5132.87993385 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚178598013042176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚663 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚187331479407104 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83347962 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8708963047470 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚178598013042176 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202791669 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚187331731091968 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49756220 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8733484367872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81424889 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31669334 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚178724199811490 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚178597995039232 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚198089732433920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚189206647012864 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8883406258688 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚4399 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202791669 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8733718049792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31668671 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202818041 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚665 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119443707 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83347962 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83347964 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81425554 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚189206623072768 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _step‚359 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81424889 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31669334 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚178724199811490 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚178597995039232 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚198089732433920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚189206647012864 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8883406258688 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚4399 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202791669 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8733718049792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31668671 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202818041 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚665 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119443707 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83347962 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83347964 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81425554 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚189206623072768 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _step‚359 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 _runtime‚ 5132.87993385 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚178598013042176 OGDeviceStatsM±Å onitor.on_train_batch_end/inactive_split.small_pool.current‚663 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚187331479407104 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83347962 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8708963047470 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚178598013042176 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202791669燲Œè2Ù- _wandbruntime‚5132  _runtime‚5132wdÚXÚÒ 1loss/FBt3/assignment_loss‚0.08587572674685069 .loss/FBt1/detection_loss‚3.137884199619293 0loss/FRt4/assignment_loss‚2.5664792919158934 /loss/FRt1/detection_loss‚2.7877334973641803 0loss/SRqqt2/detection_loss‚2.598584767045646 _step‚360 2loss/SRqqt1/assignment_loss‚1.4191732674837112 1loss/SRqqt2/assignment_loss‚1.573887949873661 0loss/FBt1/assignment_loss‚0.1414459596077601 1loss/SRqqt1/detection_loss‚1.7592491567134858 $ _timestamp‚1.788469079697612e+09 /loss/FRt1/assignment_loss‚2.650242675627981 1loss/SRqqt4/assignment_loss‚1.615417028112071 1loss/FBt2/assignment_loss‚0.09563027929609688 /loss/FRt2/detection_loss‚2.3767101684523584 .loss/FBt2/detection_loss‚2.437753850221634 /loss/FBt4/detection_loss‚2.2721695466475054 _runtime‚5132.882606292 2loss/SRqqt3/assignment_loss‚1.1679185778934467 /loss/FRt3/detection_loss‚1.6559884266081182 0loss/SRqqt4/detection_loss‚2.232225487913404 .loss/FBt3/detection_loss‚0.643156238858189 0loss/FRt3/assignment_loss‚1.2818620725111527 1loss/SRqqt3/detection_loss‚0.8335125036537647 epoch‚3 trainer/global_step‚4399 /loss/FRt2/assignment_loss‚3.558857335763819 0loss/FBt4/assignment_loss‚0.4529170557759905 /loss/FRt4/detection_loss‚3.0330321300029754 &loss/total_loss‚42.377707193708986…ºÙ ]Û× 0loss/FRt4/assignment_loss‚2.5664792919158934 /loss/FRt1/detection_loss‚2.7877334973641803 0loss/SRqqt2/detection_loss‚2.598584767045646 _step‚360 2loss/SRqqt1/assignment_loss‚1.4191732674837112 1loss/SRqqt2/assignment_loss‚1.573887949873661 0loss/FBt1/assignment_loss‚0.1414459596077601 1loss/SRqqt1/detection_loss‚1.7592491567134858 $ _timestamp‚1.788469079697612e+09 /loss/FRt1/assignment_loss‚2.650242675627981 1loss/SRqqt4/assignment_loss‚1.615417028112071 1loss/FBt2/assignment_loss‚0.09563027929609688 /loss/FRt2/detection_loss‚2.3767101684523584 .loss/FBt2/detection_loss‚2.437753850221634 /loss/FBt4/detection_loss‚2.2721695466475054 _runtime‚5132.882606292 2loss/SRqqt3/assignment_loss‚1.1679185778934467 /loss/FRt3/detection_loss‚1.6559884266081182 0loss/SRqqt4/detection_loss‚2.232225487913404 .loss/FBt3/detection_loss‚0.643156238858189 0loss/FRt3/assignment_loss‚1.2818620725111527 1loss/SRqqt3/detection_loss‚0.8335125036537647 epoch‚3 trainer/global_step‚4399 /loss/FRt2/assignment_loss‚3.558857335763819 0loss/FBt4/assignment_loss‚0.4529170557759905 /loss/FRt4/detection_loss‚3.0330321300029754 &loss/total_loss‚42.377707193708986 1loss/FBt3/assignment_loss‚0.08587572674685069 .loss/FBt1/detection_loss‚3.137884199619293è7¬I\’Ü‚(:‡ ã¾çÔÚõ²ˆ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚118.751' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.68775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vm !cÝ‚(:Ø ã¾çÔœåêˆ disk.sda1.out‚0 disk.md0.out‚ 540.390625 disk.rbd2.out‚ 1.341796875# disk.nvme0n1.out‚25256.78515625 disk.sda2.out‚ 560.265625 disk.md0p1.out‚ 540.390625 network.sent‚ 1.0352621e+07 disk.dm-0.out‚ 65697.2734375 disk.sda1.in‚0 disk.sdb2.in‚ 3.49609375 disk.md0.in‚7.125 disk.rbd2.in‚0) disk./.usagePercent‚83.18433120091909 disk.dm-0.in‚528706.6640625 disk.md0p1.in‚7.125# disk.nvme1n1.out‚40440.48828125$ disk./.usageGB‚2975.074607849121 disk.sdb.out‚ 560.265625 disk.sdb1.in‚0 disk.sda.in‚ 3.62890625 disk.sda.out‚ 560.265625* proc.memory.availableMB‚10652.33984375 proc.memory.rssMB‚3030.25) proc.memory.percent‚18.49517822265625 disk.sdb2.out‚ 560.265625 disk.sdb1.out‚0# disk.nvme0n1.in‚328311.12109375% memory_percent‚34.983277320861816 cpu‚25.14355966954931 proc.cpu.threads‚17# disk.nvme1n1.in‚200395.54296875 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 1.404296875 network.recv‚ 1.734348e+06 disk.sda2.in‚ 3.62890625 disk.sdb.in‚ 3.49609375x&ÁÒ’Þ‚(:‡ ò¾çÔæü† gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚141.087' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.27175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žs¿ê{ß‚(:ð ò¾çÔô‘ë† disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.rbd2.out‚ 1.341796875# disk.nvme1n1.out‚40495.60546875 disk.dm-0.in‚528706.6640625 proc.memory.rssMB‚3030.25 network.sent‚ 1.0447798e+07! disk.sdb2.out‚561.71728515625# disk.nvme1n1.in‚200395.54296875# disk.nvme0n1.in‚328311.12109375 disk.sda1.in‚0! disk.sda2.out‚561.71728515625% memory_percent‚34.986233711242676* proc.memory.availableMB‚10651.85546875 disk.sdb1.in‚0 disk.sda.out‚561.71728515625 disk.dm-0.out‚65885.73828125 disk.sda1.out‚0 cpu‚25.14316531195998 network.recv‚ 1.740918e+06 disk.md0.in‚7.125 disk.sda.in‚ 3.62890625" disk.nvme0n1.out‚ 25390.1328125( disk./.usagePercent‚83.1842747774976% disk./.usageGB‚2975.0725898742676 disk.md0.out‚ 541.78515625 disk.md0p1.out‚ 541.78515625) proc.memory.percent‚18.49517822265625 disk.sdb1.out‚0 disk.md0p1.in‚7.125 disk.rbd0.out‚ 1.404296875 disk.sda2.in‚ 3.62890625 disk.sdb2.in‚ 3.49609375 disk.sdb.in‚ 3.49609375 disk.sdb.out‚561.71728515625 proc.cpu.threads‚17ZŠ»‘à‚(:† ¿çÔוä‡ gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.966' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.9915 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QZ má‚(:â ¿çÔ›ïሠdisk.sda1.in‚0 disk.sda2.in‚ 3.62890625 disk.sdb2.out‚ 563.08984375# disk.nvme1n1.in‚200395.54296875 cpu‚25.14287756297838 disk.sdb.in‚ 3.49609375% disk./.usageGB‚2974.7559776306152 disk.md0p1.in‚7.125 disk.rbd0.in‚0.015625# disk.nvme1n1.out‚40673.68359375 disk.rbd0.out‚ 1.404296875 disk.sdb1.out‚0 proc.memory.rssMB‚3030.25 disk.sda1.out‚0) proc.memory.availableMB‚ 10652.3515625 proc.cpu.threads‚17 disk.sda.out‚ 563.08984375# disk.nvme0n1.in‚328311.12109375 disk.md0.out‚ 543.1015625 disk.sdb.out‚ 563.08984375 disk.dm-0.in‚528706.6640625 disk.sda2.out‚ 563.08984375 disk.sdb1.in‚0% memory_percent‚34.983205795288086 network.sent‚ 1.0453522e+07 disk.sda.in‚ 3.62890625 disk.rbd2.in‚0 disk.rbd2.out‚ 1.341796875) proc.memory.percent‚18.49517822265625) disk./.usagePercent‚83.17542216665255 disk.md0p1.out‚ 543.1015625 disk.sdb2.in‚ 3.49609375 disk.md0.in‚7.125 network.recv‚ 1.744681e+06# disk.nvme0n1.out‚25589.88671875 disk.dm-0.out‚ 66263.5703125å¼»—2â- _wandbruntime‚5187  _runtime‚5187'=p·ÇãÁ % _timestamp‚1.7884691338422742e+09 _runtime‚5187.027412208 _step‚361 & lr-AdamW/pg1‚0.0006568607489391109 & lr-AdamW/pg2‚0.0006568607489391109 trainer/global_step‚4449¸÷WÌäÆ & lr-AdamW/pg1‚0.0006568607489391109 & lr-AdamW/pg2‚0.0006568607489391109 trainer/global_step‚4449 % _timestamp‚1.7884691338422742e+09 _runtime‚5187.027412208 _step‚361é}Ñ2å- _wandbruntime‚5187  _runtime‚51872 ³mà'æÚO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚189352998572032 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚189352746900480 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8829736048640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚660 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8981245603840 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚180523010851840 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚191249161545216 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚4449 _step‚362 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84243439 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8829969717248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120781311 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚180659412897232 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32012300 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 % _timestamp‚1.7884691338434823e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 204998404 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120754967 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84243437 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318753792 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚171854470298448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120781311 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚200230088395264 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚191249137605120 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚189352998572032 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚180523028854784 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚658 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8980950790144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50285518 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82297818 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82297158 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84243437 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚180523028854784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8804942598784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚189352746900480 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8829969717248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32011642 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚171854453259088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205024750 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚200230407149056 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚180659166876788 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0H?´å'çßO GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚189352998572032 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚189352746900480 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8829736048640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚660 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8981245603840 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚180523010851840 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚191249161545216 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚4449 _step‚362 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84243439 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8829969717248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120781311 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚180659412897232 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32012300 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 % _timestamp‚1.7884691338434823e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 204998404 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120754967 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84243437 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318753792 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚171854470298448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120781311 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚200230088395264 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚191249137605120 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚189352998572032 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚180523028854784 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚658 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8980950790144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50285518 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82297818 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82297158 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84243437 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚180523028854784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8804942598784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚189352746900480 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8829969717248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32011642 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚171854453259088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205024750 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚200230407149056 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚180659166876788 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496ê,ä½2è- _wandbruntime‚5188  _runtime‚5188löz, éãM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚180698655723198 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚180562314891776 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚180562314891776 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚171891983616512 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚180698901743850 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚189394248648192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8982951445504 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _runtime‚5188.101699875 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205043371 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82314970 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32019293 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚171892000655872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84261714 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120808029 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120781659 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8983233662976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84261712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚191290845515264 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120808029 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50296320 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205069743 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚363 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚180562296888832 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8806901087978 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚189394248648192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8831933756416 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096·Úézî&êèM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚180698655723198 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚180562314891776 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚180562314891776 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚171891983616512 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚180698901743850 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚189394248648192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8982951445504 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _runtime‚5188.101699875 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205043371 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82314970 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32019293 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚171892000655872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84261714 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120808029 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120781659 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8983233662976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84261712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚191290845515264 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120808029 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50296320 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205069743 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚363 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚180562296888832 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8806901087978 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚189394248648192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8831933756416 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370ëX“$U2ë- _wandbruntime‚5188  _runtime‚5188ÁòZÜ]ì× 2loss/SRqqt3/assignment_loss‚0.8127105174741397 2loss/SRqqt4/assignment_loss‚1.7671129034715705 /loss/FBt2/detection_loss‚3.7359149008989334 .loss/FBt4/detection_loss‚1.936357468366623 epoch‚3 /loss/FRt3/detection_loss‚1.5782908670351667 % _timestamp‚1.7884691349188066e+09 1loss/FBt2/assignment_loss‚0.10395529936649836 0loss/SRqqt1/detection_loss‚1.743820030297806 trainer/global_step‚4449 0loss/FRt1/assignment_loss‚2.3312126511619207 0loss/FRt4/assignment_loss‚2.4560503473988287 .loss/FRt2/detection_loss‚2.263207522069299 /loss/FRt1/detection_loss‚2.3758254051208496 2loss/SRqqt1/assignment_loss‚1.4965430635736505 1loss/FBt1/assignment_loss‚0.22396661806487828 1loss/SRqqt4/detection_loss‚2.2561475913971663 _step‚364 1loss/FBt3/assignment_loss‚0.05080005892469313 .loss/FBt1/detection_loss‚2.709490954875946 /loss/FBt3/detection_loss‚0.8109082479409322 %loss/total_loss‚41.54979020046139 1loss/SRqqt2/assignment_loss‚1.400448909153541 0loss/FBt4/assignment_loss‚0.2280786124523729 /loss/FRt4/detection_loss‚2.8346062335703106 1loss/SRqqt2/detection_loss‚2.6501464496056237 1loss/SRqqt3/detection_loss‚1.0799199394260843 _runtime‚5188.103580583 0loss/FRt3/assignment_loss‚1.3772190789044914 0loss/FRt2/assignment_loss‚3.3270565299100654<;î7bíÜ 1loss/FBt2/assignment_loss‚0.10395529936649836 0loss/SRqqt1/detection_loss‚1.743820030297806 trainer/global_step‚4449 0loss/FRt1/assignment_loss‚2.3312126511619207 0loss/FRt4/assignment_loss‚2.4560503473988287 .loss/FRt2/detection_loss‚2.263207522069299 /loss/FRt1/detection_loss‚2.3758254051208496 2loss/SRqqt1/assignment_loss‚1.4965430635736505 1loss/FBt1/assignment_loss‚0.22396661806487828 1loss/SRqqt4/detection_loss‚2.2561475913971663 _step‚364 1loss/FBt3/assignment_loss‚0.05080005892469313 .loss/FBt1/detection_loss‚2.709490954875946 /loss/FBt3/detection_loss‚0.8109082479409322 %loss/total_loss‚41.54979020046139 1loss/SRqqt2/assignment_loss‚1.400448909153541 0loss/FBt4/assignment_loss‚0.2280786124523729 /loss/FRt4/detection_loss‚2.8346062335703106 1loss/SRqqt2/detection_loss‚2.6501464496056237 1loss/SRqqt3/detection_loss‚1.0799199394260843 _runtime‚5188.103580583 0loss/FRt3/assignment_loss‚1.3772190789044914 0loss/FRt2/assignment_loss‚3.3270565299100654 2loss/SRqqt3/assignment_loss‚0.8127105174741397 2loss/SRqqt4/assignment_loss‚1.7671129034715705 /loss/FBt2/detection_loss‚3.7359149008989334 .loss/FBt4/detection_loss‚1.936357468366623 epoch‚3 /loss/FRt3/detection_loss‚1.5782908670351667 % _timestamp‚1.7884691349188066e+09ì©^ß‘î‚(:† ¿çÔËÐñ‰ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.966' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.9915 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0T2rï‚(:ç ¿çÔáù¾) disk./.usagePercent‚83.17658636636067 disk.md0.out‚ 545.03125 disk.sdb1.in‚0 disk.sda1.out‚0 disk.md0p1.out‚ 545.03125 network.sent‚ 1.045883e+07 disk.rbd2.in‚0 disk.dm-0.out‚66439.51171875 disk.sda.in‚ 3.62890625 disk.sda.out‚565.0791015625* proc.memory.availableMB‚10652.10546875 proc.cpu.threads‚17# disk.nvme0n1.in‚328311.12109375 disk.sda2.out‚565.0791015625% memory_percent‚34.984707832336426% disk./.usageGB‚2974.7976150512695 disk.sdb2.out‚565.0791015625# disk.nvme0n1.out‚25722.87890625 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 1.341796875 disk.sdb.out‚565.0791015625 disk.sda2.in‚ 3.62890625 disk.sdb2.in‚ 3.49609375 disk.md0.in‚7.125 disk.rbd0.out‚ 1.404296875" disk.nvme1n1.out‚ 40716.6328125 disk.sdb.in‚ 3.49609375 disk.sdb1.out‚0 disk.dm-0.in‚528706.6640625 proc.memory.rssMB‚3030.25# disk.nvme1n1.in‚200395.54296875 disk.sda1.in‚0 disk.md0p1.in‚7.125) proc.memory.percent‚18.49517822265625 cpu‚25.142564469298566 network.recv‚ 1.748884e+06€óoýð‚(:… Ÿ¿çÔ¥Ó©† gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚142.036' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.509 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯ÂLrñ‚(:ç Ÿ¿çÔŒ·Û‡ disk.md0.in‚7.125 disk.rbd2.out‚ 1.341796875 disk.sdb.in‚ 3.49609375 disk.dm-0.in‚529150.06640625 disk.dm-0.out‚ 66596.921875 disk.md0p1.in‚7.125 proc.memory.rssMB‚3030.25 cpu‚25.14226075463528 network.recv‚ 1.755162e+06! disk.sdb2.out‚566.68603515625# disk.nvme1n1.out‚40717.31640625 disk.sdb1.in‚0 disk.sda.out‚566.68603515625 disk.sda2.in‚ 3.62890625) disk./.usagePercent‚83.17722824944298 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.sda.in‚ 3.62890625 disk.md0p1.out‚ 546.578125 proc.cpu.threads‚17$ disk./.usageGB‚2974.820571899414 disk.sdb2.in‚ 3.49609375# disk.nvme0n1.out‚25879.60546875$ memory_percent‚34.98499393463135 disk.rbd0.out‚ 1.404296875# disk.nvme1n1.in‚200566.37890625 disk.sdb.out‚566.68603515625 disk.nvme0n1.in‚ 328583.6875 disk.sda1.out‚0 network.sent‚ 1.0567767e+07 disk.md0.out‚ 546.578125 disk.rbd0.in‚0.015625 disk.sda1.in‚0* proc.memory.availableMB‚10652.05859375! disk.sda2.out‚566.68603515625) proc.memory.percent‚18.49517822265625|"w‘ò‚(:† ®¿çÔ„Ñ¢† gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚107.002' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚26.7505 gpu.0.smClock‚1200 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ]»Ýoó‚(:ä ®¿çÔ·ñ¯‡ network.sent‚ 1.0571842e+07 disk.md0.in‚7.125 disk.md0p1.in‚7.125 network.recv‚ 1.759837e+06 disk.sdb.out‚568.00927734375 disk.nvme0n1.in‚ 328583.6875 disk.dm-0.out‚ 66803.0234375& proc.memory.availableMB‚ 10651.5625 proc.cpu.threads‚17% disk./.usageGB‚2974.8538398742676 disk.rbd0.out‚ 1.404296875# disk.nvme1n1.out‚40717.81640625 disk.sdb.in‚ 3.49609375 disk.sdb1.out‚0# disk.nvme0n1.out‚26085.20703125 disk.sda1.in‚0 disk.sda.out‚568.00927734375 disk.sda2.in‚ 3.62890625 proc.memory.rssMB‚3030.25 disk.rbd2.in‚0 disk.sdb2.in‚ 3.49609375! disk.sdb2.out‚568.00927734375# disk.nvme1n1.in‚200566.37890625 disk.sdb1.in‚0! disk.sda2.out‚568.00927734375$ memory_percent‚34.98802185058594) proc.memory.percent‚18.49517822265625 disk.rbd2.out‚ 1.341796875 disk.dm-0.in‚529150.06640625 cpu‚25.14195306096835 disk.md0p1.out‚ 547.859375( disk./.usagePercent‚83.1781584359436 disk.md0.out‚ 547.859375 disk.rbd0.in‚0.015625 disk.sda.in‚ 3.62890625 disk.sda1.out‚0j7° ‘ô‚(:† ½¿çԵ椆 gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.357' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.83925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ki`Úoõ‚(:ä ½¿çÔ‘çú† disk.sda1.out‚0 disk.sdb2.out‚ 570.158203125 proc.memory.rssMB‚3030.25 cpu‚25.141647974653296# disk.nvme1n1.out‚40717.94140625 disk.nvme0n1.in‚ 328583.6875 disk.dm-0.in‚529150.06640625 disk.sda1.in‚0) proc.memory.percent‚18.49517822265625% disk./.usageGB‚2975.0720291137695# disk.nvme1n1.in‚200566.37890625" disk.nvme0n1.out‚ 26386.3203125 disk.sda2.out‚ 570.158203125 disk.sdb2.in‚ 3.49609375$ memory_percent‚34.98647212982178 network.sent‚ 1.0577647e+07) disk./.usagePercent‚83.18425909839937 disk.rbd0.in‚0.015625 disk.sdb1.in‚0 disk.md0p1.in‚7.125 disk.rbd2.in‚0 disk.rbd2.out‚ 1.341796875 disk.sdb1.out‚0 disk.md0p1.out‚ 549.9296875 network.recv‚ 1.764039e+06 disk.rbd0.out‚ 1.404296875 disk.sda.out‚ 570.158203125 disk.md0.in‚7.125 disk.sda.in‚ 3.62890625 disk.dm-0.out‚67104.26171875 disk.sda2.in‚ 3.62890625 disk.md0.out‚ 549.9296875 disk.sdb.in‚ 3.49609375* proc.memory.availableMB‚10651.81640625 proc.cpu.threads‚17 disk.sdb.out‚ 570.158203125_£—¤2ö- _wandbruntime‚5242  _runtime‚5242â¾jÇ÷Á _step‚365 & lr-AdamW/pg1‚0.0006520456585087117 & lr-AdamW/pg2‚0.0006520456585087117 trainer/global_step‚4499 % _timestamp‚1.7884691892427475e+09 _runtime‚5242.427838461cš8ˆÌøÆ & lr-AdamW/pg1‚0.0006520456585087117 & lr-AdamW/pg2‚0.0006520456585087117 trainer/global_step‚4499 % _timestamp‚1.7884691892427475e+09 _runtime‚5242.427838461 _step‚365íº2ù- _wandbruntime‚5242  _runtime‚5242ÜØ¡]êúÚO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚193333336107520 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚182487312701440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 % _timestamp‚1.7884691892439277e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚182487330704384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50825618 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9081032588800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85157189 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122092919 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83187591 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚191415264456704 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚173730988169648 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8902651658208 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _runtime‚5242.429378138 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85157189 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚173730971130288 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251,,°‚ö671552 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚683 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9080744066560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚182633622788496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 122119263 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8928185423872 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83188274 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚202414392636416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚191415516128256 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 _step‚366 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚191415516128256 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚191415264456704 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8927951755264 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚182487312701440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 122119263 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚182487330704384 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚191415264456704 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚173730988169648 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8902651658208 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _runtime‚5242.429378138 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85157189 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚173730971130288 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚683 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9080744066560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚182633622788496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 122119263 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8928185423872 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83188274 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚202414392636416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚191415516128256 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 _step‚366 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚191415516128256 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚191415264456704 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8927951755264 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚182487312701440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 122119263 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚182487330704384 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83205027 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8929915781120 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85175462 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32369264 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8930149463040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8904610147194 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚182526616741376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83205684 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚202458071907328 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207295073 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4499 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85175464 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚191456766204416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 122145981 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚657 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚191456514519552 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚182526616741376 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50836420 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50836418 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207321445 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚182526598738432 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122119611 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚655 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 122119611 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9083027889664 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _step‚367 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚173768501487712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8904839128486 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9082735186432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85175462 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8929915781120 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32368609 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚193375020077568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85175464 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8904839128486 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9082735186432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85175462 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8929915781120 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32368609 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚193375020077568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85175464 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207295073 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83205027 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8929915781120 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85175462 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32369264 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8930149463040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8904610147194 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚182526616741376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83205684 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚202458071907328 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207295073 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4499 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85175464 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚191456766204416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 122145981 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚657 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚191456514519552 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚182526616741376 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50836420 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50836418 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207321445 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚182526598738432 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122119611 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚655 D>DevicAÒÄ eStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 122119611 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9083027889664 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _step‚367 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚173768501487712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚175607489001488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚193478033684480 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚702 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9026401130496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9000589698716 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 _step‚370 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚184451614551040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚204598432265216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9180873715200 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚184608078700204 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚184451632553984 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚184451614551040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚193477782012928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 209528154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86070939 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51365718 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51365716 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 329239552 # _timestamp‚1.78846924057199e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9000818679800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32706416 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚195417558550016 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚175607489001488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚193478033684480 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚702 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9026401130496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9000589698716 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 _step‚370 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚184451614551040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚204598432265216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9180873715200 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚184608078700204 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚184451632553984 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚184451614551040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚193477782012928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 209528154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86070939 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51365718 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51365716 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 329239552 # _timestamp‚1.78846924057199e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9000818679800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32706416 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚195417558550016 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209573147 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32713434 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9182567201280 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 123457563 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86089214 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86089212 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 123483933 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 4.DeviceStatsMonitor.on_tN§5_rain_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚650 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚184490900588032 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚193519032075776 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84089302 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 209573147 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚184490918590976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84089954 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚204642100327424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32712784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚204641785781248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚184647813567266 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚195459242520064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86089212 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚184490900588032 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86089214 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚195459218579968 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9002777168994 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚371 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51376520 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9182857807360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51376518`”ô`ï&éM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 123457563 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86089214 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86089212 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 123483933 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚650 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚184490900588032 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚193519032075776 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84089302 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 209573147 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚184490918590976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84089954 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚204642100327424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32712784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚204641785781248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚184647813567266 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚195459242520064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86089212 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚184490900588032 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86089214 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚195459218579968 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9002777168994 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚371 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51376520 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9182857807360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51376518 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 123483933 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚193519283760640 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209573147 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32713434 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9182567201280óc‰K­2‘- _wandbruntime‚5294  _runtime‚5294¿mÀW’Ñ 0loss/FRt1/assignment_loss‚2.8533380706059304 0loss/FRt2/assignment_loss‚3.3883130091887255 1loss/SRqqt4/assignment_loss‚1.862048477679491 /loss/FBt1/assignment_loss‚0.313262545526959 0loss/FBt4/assignment_loss‚0.7286822540112413 $ _timestamp‚1.788469241673428e+09 .loss/FBt3/detection_loss‚0.766946795099001 1loss/FBt2/assignment_loss‚0.11419508803230481 /loss/FRt4/detection_loss‚3.0003185868263245 /loss/FRt1/detection_loss‚2.5706645924794045 1loss/SRqqt3/detection_loss‚0.8752711977385268 .loss/FBt1/detection_loss‚2.802224540710449 _runtime‚5294.858540153 /loss/FRt4/assignment_loss‚3.062210348936228 2loss/SRqqt1/assignment_loss‚1.5446717855977077 0loss/FBt3/assignment_loss‚0.0167322660348385 1loss/SRqqt2/detection_loss‚2.4216813598909686 /loss/FBt2/detection_loss‚2.2913773013995242 epoch‚3 2loss/SRqqt2/assignment_loss‚1.7852789164993033 .loss/FRt3/detection_loss‚1.358544822682937 %loss/total_loss‚42.92759405978872 0loss/FRt3/assignment_loss‚1.5894319264094035 2loss/SRqqt3/assignment_loss‚0.9668566922928337 /loss/FRt2/detection_loss‚2.3218361681792885 .loss/FBt4/detection_loss‚1.843855809706908 trainer/global_step‚4549 1loss/SRqqt1/detection_loss‚1.9588186644455965 0loss/SRqqt4/detection_loss‚2.491032839814822 _step‚372;YŒO\“Ö 0loss/FRt1/assignment_loss‚2.8533380706059304 0loss/FRt2/assignment_loss‚3.3883130091887255 1loss/SRqqt4/assignment_loss‚1.862048477679491 /loss/FBt1/assignment_loss‚0.313262545526959 0loss/FBt4/assignment_loss‚0.7286822540112413 $ _timestamp‚1.788469241673428e+09 .loss/FBt3/detection_loss‚0.766946795099001 1loss/FBt2/assignment_loss‚0.11419508803230481 /loss/FRt4/detection_loss‚3.0003185868263245 /loss/FRt1/detection_loss‚2.5706645924794045 1loss/SRqqt3/detection_loss‚0.8752711977385268 .loss/FBt1/detection_loss‚2.802224540710449 _runtime‚5294.858540153 /loss/FRt4/assignment_loss‚3.062210348936228 2loss/SRqqt1/assignment_loss‚1.5446717855977077 0loss/FBt3/assignment_loss‚0.0167322660348385 1loss/SRqqt2/detection_loss‚2.4216813598909686 /loss/FBt2/detection_loss‚2.2913773013995242 epoch‚3 2loss/SRqqt2/assignment_loss‚1.7852789164993033 .loss/FRt3/detection_loss‚1.358544822682937 %loss/total_loss‚42.92759405978872 0loss/FRt3/assignment_loss‚1.5894319264094035 2loss/SRqqt3/assignment_loss‚0.9668566922928337 /loss/FRt2/detection_loss‚2.3218361681792885 .loss/FBt4/detection_loss‚1.843855809706908 trainer/global_step‚4549 1loss/SRqqt1/detection_loss‚1.9588186644455965 0loss/SRqqt4/detection_loss‚2.491032839814822 _step‚372ôAØÔS›”‚(: ù¿çÔ¥·Ë‡ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M·‹/y•‚(:î ù¿çÔܜ͈ disk.rbd0.out‚ 1.404296875 disk.sdb.in‚ 3.49609375 disk.sdb.out‚576.67431640625 disk.sda.out‚576.67431640625 disk.sda2.in‚ 3.62890625# disk.nvme1n1.in‚200566.37890625 disk.sdb2.in‚ 3.49609375 disk.dm-0.in‚ 529187.140625 disk.md0p1.out‚ 556.19921875 proc.memory.rssMB‚3030.25) proc.memory.percent‚18.49517822265625 disk.sdb1.out‚0 disk.sda1.in‚0 disk.md0.in‚7.125( proc.memory.availableMB‚ 10652.015625 disk.rbd0.in‚0.015625# disk.nvme0n1.in‚328620.76171875# disk.nvme0n1.out‚27021.17578125 disk.sda1.out‚0 proc.cpu.threads‚17 disk.dm-0.out‚ 67739.7109375! disk.sda2.out‚576.67431640625 network.sent‚ 1.0702583e+07% disk./.usageGB‚2973.6962280273438! disk.sdb2.out‚576.67431640625 disk.md0.out‚ 556.19921875 disk.sdb1.in‚0 disk.md0p1.in‚7.125) disk./.usagePercent‚83.14579112420543 disk.rbd2.out‚ 1.341796875# disk.nvme1n1.out‚40718.53515625$ memory_percent‚34.98525619506836 cpu‚25.140318409688682 network.recv‚ 1.783285e+06 disk.rbd2.in‚0 disk.sda.in‚ 3.62890625[÷X¨’–‚(:‡ ˆÀçÔ¬¾½‡ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚116.583' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.14575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:ˆ•xd—‚(:Ù ˆÀçÔ§±õ‡) disk./.usagePercent‚83.14448517265014 disk.sdb2.in‚ 3.49609375 disk.dm-0.in‚ 529187.140625# disk.nvme0n1.in‚328620.76171875 disk.sda1.in‚0 proc.cpu.threads‚17" disk.nvme1n1.out‚ 40719.0546875# disk.nvme1n1.in‚200566.37890625 disk.sdb.out‚ 578.451171875 disk.md0p1.out‚ 557.90625) proc.memory.percent‚18.49517822265625 cpu‚25.13997783422479 network.recv‚1.79e+06 disk.sdb2.out‚ 578.451171875 disk.md0.in‚7.125 disk.sda2.in‚ 3.62890625 disk.sda2.out‚ 578.451171875 network.sent‚ 1.0812448e+07% disk./.usageGB‚2973.6495208740234 disk.rbd0.out‚ 1.404296875 disk.rbd2.out‚ 1.341796875 disk.sdb1.out‚0 disk.md0p1.in‚7.125 proc.memory.rssMB‚3030.25 disk.md0.out‚ 557.90625 disk.rbd0.in‚0.015625 disk.sdb1.in‚0 disk.sda.out‚ 578.451171875 disk.dm-0.out‚ 67810.28125 disk.sda1.out‚0* proc.memory.availableMB‚10652.21484375 disk.rbd2.in‚0 disk.sda.in‚ 3.62890625$ memory_percent‚34.98404026031494 disk.sdb.in‚ 3.49609375" disk.nvme0n1.out‚ 27091.2265625G;Ò—‘˜‚(:† —ÀçÔÜΧ‡ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚116.306' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.0765 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë´_ n™‚(:ã —ÀçÔ»Ïÿ‡ disk.md0.in‚7.125 disk.sdb.out‚579.8154296875$ memory_percent‚34.98556613922119 disk.md0p1.out‚ 559.21875 disk.sdb2.out‚579.8154296875 disk.rbd2.in‚0# disk.nvme1n1.out‚40719.12109375) disk./.usagePercent‚83.15268096793835 disk.rbd0.out‚ 1.404296875 disk.sdb.in‚ 3.49609375 disk.dm-0.in‚ 529187.140625 disk.sda2.in‚ 3.62890625 disk.sda2.out‚579.8154296875) proc.memory.percent‚18.49517822265625 proc.cpu.threads‚17# disk.nvme1n1.in‚200566.37890625 disk.sda.out‚579.8154296875 cpu‚25.1397251824811 disk.sdb1.in‚0 disk.sdb1.out‚0" disk.nvme0n1.out‚ 27252.8671875$ disk./.usageGB‚2973.942642211914 disk.sdb2.in‚ 3.49609375 disk.md0.out‚ 559.21875# disk.nvme0n1.in‚328620.76171875 disk.dm-0.out‚67971.98828125* proc.memory.availableMB‚10651.96484375 network.sent‚ 1.0815568e+07 network.recv‚ 1.793631e+06 disk.rbd2.out‚ 1.341796875 disk.sda.in‚ 3.62890625 disk.sda1.out‚0 disk.md0p1.in‚7.125 disk.rbd0.in‚0.015625 disk.sda1.in‚0 proc.memory.rssMB‚3030.25mu °š‚(:… ¦ÀçÔôƒ¶‡ gpu.0.gpu‚26.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚124.044' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.011 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0iè€p›‚(:å ¦ÀçÔóîÛ‡ disk.md0.out‚ 560.96875 disk.nvme0n1.out‚ 27392.90625% disk./.usageGB‚2974.0683250427246 disk.rbd2.out‚ 1.341796875# disk.nvme1n1.in‚200566.37890625 disk.sdb.in‚ 3.49609375 proc.memory.rssMB‚3030.25) proc.memory.percent‚18.49517822265625 proc.cpu.threads‚17 disk.sda1.out‚0 network.recv‚ 1.798342e+06 disk.md0.in‚7.125 disk.sdb1.in‚0 disk.sda.in‚ 3.62890625 disk.dm-0.out‚68112.24609375 disk.sda1.in‚0 disk.md0p1.in‚7.125 cpu‚25.139430993411302) disk./.usagePercent‚83.15619511249017! disk.sdb2.out‚581.63232421875 disk.rbd2.in‚0 disk.sdb.out‚581.63232421875 disk.sda.out‚581.63232421875# disk.nvme0n1.in‚328620.76171875 network.sent‚ 1.0821982e+07 disk.sdb1.out‚0 disk.sda2.in‚ 3.62890625 disk.md0p1.out‚ 560.96875$ memory_percent‚34.98859405517578 disk.sdb2.in‚ 3.49609375 disk.rbd0.out‚ 1.404296875 disk.dm-0.in‚ 529187.140625' proc.memory.availableMB‚ 10651.46875! disk.sda2.out‚581.63232421875 disk.rbd0.in‚0.015625# disk.nvme1n1.out‚40719.33984375xŸ2î2œ- _wandbruntime‚5347  _runtime‚5347åvÌÇÁ & lr-AdamW/pg1‚0.0006422665575682188 & lr-AdamW/pg2‚0.0006422665575682188 trainer/global_step‚4599 % _timestamp‚1.7884692947689111e+09 _runtime‚5347.953886389 _step‚373ÓžŒqÌžÆ & lr-AdamW/pg1‚0.0006422665575682188 & lr-AdamW/pg2‚0.0006422665575682188 trainer/global_step‚4599 % _timestamp‚1.7884692947689111e+09 _runtime‚5347.953886389 _step‚373õ 諤2Ÿ- _wandbruntime‚5347  _runtime‚5347 ÝÚO ÚO OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9280676012032 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚186582534611912 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86984687 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚195540551240704 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚186415934403584 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚197501757052416 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9098756720308 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884692947699485e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚195540551240704 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 124795167 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚195540299569152 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚186415934403584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚688 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚177484023912048 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9124616837120 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚197501733112320 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚186582780632356 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84959995 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚206782118504960 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚177484006872688 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _step‚374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚690 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9124383168512 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86984687 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33054867 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 211779856 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚206782433064448 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9098527739224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚186415916400640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9124383168512 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86984689 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0K¶îå'¡ßO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84960685 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124795167 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚195540299569152 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚186415934403584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚688 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚177484023912048 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9124616837120 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚197501733112320 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚186582780632356 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84959995 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚206782118504960 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚177484006872688 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _step‚374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚690 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9124383168512 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86984687 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33054867 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 211779856 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚206782433064448 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9098527739224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚186415916400640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9124383168512 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86984689 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9280676012032 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚186582534611912 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86984687 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚195540551240704 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚186415934403584 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚197501757052416 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9098756720308 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884692947699485e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚195540551240704 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 124795167 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚186622269478974 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9126580876288 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚186455220440576 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84977305 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚195581801316864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9126580876288 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211824849 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚195581801316864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚197543417082368 _step‚375 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚197543441022464 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 211824849 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 211798477 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33061389 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124821885 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚206826110270464 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚177521537230112 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9100486228210 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 $ _timestamp‚1.788469295724426e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9282669248000 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9100715209502 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87002962 P–O'L ¤èM E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9100486228210 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 $ _timestamp‚1.788469295724426e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9282669248000 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9100715209502 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87002962 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚186622269478974 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9126580876288 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚186455220440576 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84977305 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚195581801316864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9126580876288 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211824849 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚199#Ï8¢5581801316864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚197543417082368 _step‚375 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚197543441022464 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 211824849 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 211798477 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33061389 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124821885 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚206826110270464 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚177521537230112 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0÷Ò^2¥- _wandbruntime‚5348  _runtime‚5348g¤ÕñW¦Ñ 0loss/SRqqt1/detection_loss‚1.761454447000115 trainer/global_step‚4599 _step‚376 1loss/FBt4/assignment_loss‚0.14408557355454832 /loss/FRt3/detection_loss‚1.4540018764707971 % _timestamp‚1.7884692957266448e+09 .loss/FRt4/detection_loss‚3.149478495121002 .loss/FBt2/detection_loss‚3.066172771155834 1loss/SRqqt2/assignment_loss‚2.547524758884984 1loss/SRqqt2/detection_loss‚2.2923643512110554 1loss/SRqqt3/detection_loss‚0.8160935341863029 epoch‚3 0loss/FBt2/assignment_loss‚0.3077622251585126 /loss/FBt1/detection_loss‚2.3645454943180084 /loss/FRt2/assignment_loss‚3.890777494849228 2loss/SRqqt1/assignment_loss‚1.3351073903893982 1loss/SRqqt3/assignment_loss‚1.140221102494208 1loss/FBt1/assignment_loss‚0.09636459132889286 _runtime‚5348.911357531 .loss/FRt3/assignment_loss‚1.37421875315554 /loss/FBt3/detection_loss‚0.5897847802520523 /loss/FRt1/detection_loss‚2.7653929851949215 /loss/FRt4/assignment_loss‚2.977592436330659 1loss/SRqqt4/assignment_loss‚1.589446994359605 1loss/FBt3/assignment_loss‚0.01653434745807658 1loss/SRqqt4/detection_loss‚2.2978483010083437 /loss/FBt4/detection_loss‚3.2705584863821664 $loss/total_loss‚44.5694543585533 0loss/FRt1/assignment_loss‚3.0468089375644922 /loss/FRt2/detection_loss‚2.2753142307245557ÁÂf\§Ö 1loss/SRqqt2/assignment_loss‚2.547524758884984 1loss/SRqqt2/detection_loss‚2.2923643512110554 1loss/SRqqt3/detection_loss‚0.8160935341863029 epoch‚3 0loss/FBt2/assignment_loss‚0.3077622251585126 /loss/FBt1/detection_loss‚2.3645454943180084 /loss/FRt2/assignment_loss‚3.890777494849228 2loss/SRqqt1/assignment_loss‚1.3351073903893982 1loss/SRqqt3/assignment_loss‚1.140221102494208 1loss/FBt1/assignment_loss‚0.09636459132889286 _runtime‚5348.911357531 .loss/FRt3/assignment_loss‚1.37421875315554 /loss/FBt3/detection_loss‚0.5897847802520523 /loss/FRt1/detection_loss‚2.7653929851949215 /loss/FRt4/assignment_loss‚2.977592436330659 1loss/SRqqt4/assignment_loss‚1.589446994359605 1loss/FBt3/assignment_loss‚0.01653434745807658 1loss/SRqqt4/detection_loss‚2.2978483010083437 /loss/FBt4/detection_loss‚3.2705584863821664 $loss/total_loss‚44.5694543585533 0loss/FRt1/assignment_loss‚3.0468089375644922 /loss/FRt2/detection_loss‚2.2753142307245557 0loss/SRqqt1/detection_loss‚1.761454447000115 trainer/global_step‚4599 _step‚376 1loss/FBt4/assignment_loss‚0.14408557355454832 /loss/FRt3/detection_loss‚1.4540018764707971 % _timestamp‚1.7884692957266448e+09 .loss/FRt4/detection_loss‚3.149478495121002 .loss/FBt2/detection_loss‚3.066172771155834øu! Ó‘¨‚(:† µÀçÔìœú‡ gpu.0.gpu‚8.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚145.623' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.40575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/‡DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85845297 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87898437 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚208966510469632 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884693497822046e+09 _runtime‚5402.967288818 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9380266392576 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚378 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33399381 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9196465779732 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52445918 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126133119 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚179360541783248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚197602817125376 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚648 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚208966198007296 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚188380218250240 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126106775 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚197602817125376 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9196465779732 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52445918 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126133119 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚179360541783248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚197602817125376 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚648 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚208966198007296 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚188380218250240 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126106775 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚197602817125376 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85845297 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87898437 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚208966510469632 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884693497822046e+09 _runtime‚5402.967288818 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532ú€Ò™?2¶- _wandbruntime‚5403  _runtime‚5403EùзãM QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚188419504287232 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126159837 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85862762 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚197644318873088 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚188419522290176 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚199627615584768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 $ _timestamp‚1.788469350763335e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52456720 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚209009879060992 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214076551 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚188596725390682 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9224562900992 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚179398072140672 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214076551 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9224796582912 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9382263476224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚4649 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33406698 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87916714 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87916712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚197644067188224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9382549888000 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9198424268718 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 126133467 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚188596479370030 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9224796582912 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚654 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 214050179 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚197644318873088 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚4649 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33406698 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87916714 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87916712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚197644067188224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9382549888000 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9198424268718 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 126133467 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚188596479370030 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9224796582912 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚654 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 214050179 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚197644318873088 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚188419504287232 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126159837 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85862762 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚197644318873088 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚188419522290176 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚199627615584768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 $ _timestamp‚1.788469350763335e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52456720 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚209009879060992 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214076551 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚188596725390682 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9224562900992 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚179398072140672ûe~„2¹- _wandbruntime‚5403  _runtime‚5403–ÐÁZºÔ 0loss/FBt2/assignment_loss‚0.2844445965674822 .loss/FRt1/detection_loss‚2.965576546390851 1loss/SRqqt4/detection_loss‚1.9610303461551666 ,loss/FRt3/detection_loss‚1.6445763598755 1loss/SRqqt2/detection_loss‚2.7671442437171936 %loss/total_loss‚44.18355724637918 % _timestamp‚1.7884693507651503e+09 /loss/FRt2/assignment_loss‚4.019081882402008 /loss/FRt4/assignment_loss‚2.691292152404785 2loss/FBt1/assignment_loss‚0.005334688311450009 0loss/FBt4/assignment_loss‚0.1251505202666811 /loss/FRt2/detection_loss‚1.8788855604042172 /loss/FRt4/detection_loss‚3.0417805886268616 .loss/FBt3/detection_loss‚0.694681776747203 /loss/FBt4/detection_loss‚2.6692616289312188 2loss/SRqqt3/assignment_loss‚0.9353463012492285 1loss/FBt3/assignment_loss‚0.05965910689975719 epoch‚3 _step‚380 /loss/FRt1/assignment_loss‚3.271302358309428 2loss/SRqqt1/assignment_loss‚1.6158038932451253 2loss/SRqqt2/assignment_loss‚2.0880026327073575 _runtime‚5403.949895582 0loss/FRt3/assignment_loss‚1.3802988135814667 1loss/SRqqt3/detection_loss‚0.8774064533412457 /loss/FBt1/detection_loss‚2.9936116422925676 /loss/FBt2/detection_loss‚2.7876081705093383 2loss/SRqqt4/assignment_loss‚1.7674894601572304 1loss/SRqqt1/detection_loss‚1.6587875232858174 trainer/global_step‚46492>~E_»Ù 0loss/FBt2/assignment_loss‚0.2844445965674822 .loss/FRt1/detection_loss‚2.965576546390851 1loss/SRqqt4/detection_loss‚1.9610303461551666 ,loss/FRt3/detection_loss‚1.6445763598755 1loss/SRqqt2/detection_loss‚2.7671442437171936 %loss/total_loss‚44.18355724637918 % _timestamp‚1.7884693507651503e+09 /loss/FRt2/assignment_loss‚4.019081882402008 /loss/FRt4/assignment_loss‚2.691292152404785 2loss/FBt1/assignment_loss‚0.005334688311450009 0loss/FBt4/assignment_loss‚0.1251505202666811 /loss/FRt2/detection_loss‚1.8788855604042172 /loss/FRt4/detection_loss‚3.0417805886268616 .loss/FBt3/detection_loss‚0.694681776747203 /loss/FBt4/detection_loss‚2.6692616289312188 2loss/SRqqt3/assignment_loss‚0.9353463012492285 1loss/FBt3/assignment_loss‚0.05965910689975719 epoch‚3 _step‚380 /loss/FRt1/assignment_loss‚3.271302358309428 2loss/SRqqt1/assignment_loss‚1.6158038932451253 2loss/SRqqt2/assignment_loss‚2.0880026327073575 _runtime‚5403.949895582 0loss/FRt3/assignment_loss‚1.3802988135814667 1loss/SRqqt3/detection_loss‚0.8774064533412457 /loss/FBt1/detection_loss‚2.9936116422925676 /loss/FBt2/detection_loss‚2.7876081705093383 2loss/SRqqt4/assignment_loss‚1.7674894601572304 1loss/SRqqt1/detection_loss‚1.6587875232858174 trainer/global_step‚4649üU¹ÈY’¼‚(:‡ ñÀçԾЃ‡ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚136.743' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.18575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ã¨ô h½‚(:Ý ñÀçԻ¨‡ disk.sdb1.out‚0% memory_percent‚34.987759590148926* proc.memory.availableMB‚10651.60546875 disk.sdb2.in‚ 3.49609375# disk.nvme1n1.out‚40720.23046875 disk.sda.out‚589.7646484375 disk.sda1.in‚0 disk.md0p1.in‚7.125 disk.md0p1.out‚568.8125 disk.rbd2.in‚0 disk.sdb1.in‚0# disk.nvme0n1.out‚28324.55078125 disk.sda2.in‚ 3.62890625) proc.memory.percent‚18.49517822265625 disk.sda1.out‚0 disk.rbd2.out‚ 1.341796875) disk./.usagePercent‚83.17821101958594% disk./.usageGB‚2974.8557205200195 disk.sdb2.out‚589.7646484375 disk.md0.in‚7.125 disk.nvme1n1.in‚ 200722.9375 disk.sda.in‚ 3.62890625 disk.rbd0.out‚ 1.404296875! disk.nvme0n1.in‚ 328970.015625 disk.dm-0.in‚ 529692.953125 disk.dm-0.out‚ 69044.78125 disk.sda2.out‚589.7646484375 proc.memory.rssMB‚3030.25 cpu‚25.138020174302895 proc.cpu.threads‚17 disk.rbd0.in‚0.015625 network.sent‚ 1.0952546e+07 disk.md0.out‚568.8125 disk.sdb.in‚ 3.49609375 disk.sdb.out‚589.7646484375 network.recv‚ 1.821394e+06²'ä’¾‚(:‡ €ÁçÔ×Ô̇ gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚119.701' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.92525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Tì«i¿‚(:Þ €ÁçÔÀÚˆˆ disk.sda.in‚ 3.62890625 disk.rbd0.out‚ 1.404296875 cpu‚25.137771322914073 disk.nvme1n1.in‚ 200722.9375 disk.sda1.out‚0 disk.sda2.in‚ 3.62890625 disk.sda2.out‚591.7490234375 disk.sdb2.in‚ 3.49609375 disk.sdb2.out‚591.7490234375% disk./.usageGB‚2974.9489707946777 disk.sdb.in‚ 3.49609375 disk.dm-0.in‚ 529692.953125 disk.sda1.in‚0 disk.rbd2.in‚0 proc.memory.rssMB‚3030.25 proc.cpu.threads‚17 network.recv‚ 1.828449e+06! disk.nvme0n1.in‚ 328970.015625 disk.md0.in‚7.125) proc.memory.percent‚18.49517822265625 network.sent‚ 1.1062622e+07 disk.sdb1.in‚0" disk.nvme0n1.out‚ 28575.2421875 disk.dm-0.out‚ 69295.96875 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 1.341796875$ memory_percent‚34.98847484588623) disk./.usagePercent‚83.18081833629365" disk.nvme1n1.out‚ 40720.7265625 disk.md0p1.out‚ 570.734375 disk.md0.out‚ 570.734375 disk.sdb1.out‚0 disk.sda.out‚591.7490234375 disk.md0p1.in‚7.125* proc.memory.availableMB‚10651.48828125 disk.sdb.out‚591.7490234375kUÉ›À‚(: ÁçÔ¢† gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.17' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚27.292500000000004 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ôk_‡eÁ‚(:Ú ÁçÔÊ‘ä‡ disk.sda2.out‚ 593.498046875 cpu‚25.137488159564654) disk./.usagePercent‚83.18430592237299 disk.sdb.in‚ 3.49609375 disk.sdb.out‚ 593.498046875 disk.sda.in‚ 3.62890625 disk.dm-0.out‚69556.09765625 disk.sda2.in‚ 3.62890625% memory_percent‚34.987831115722656' proc.memory.availableMB‚ 10651.59375 disk.nvme1n1.out‚ 40720.8125 disk.sdb2.out‚ 593.498046875 disk.md0.out‚ 572.4140625 proc.memory.rssMB‚3030.25 proc.cpu.threads‚17 network.sent‚ 1.1065547e+07 disk.sdb1.in‚0 disk.sda.out‚ 593.498046875 disk.md0.in‚7.125 disk.dm-0.in‚ 529692.953125 disk.sda1.in‚0 disk.md0p1.in‚7.125 disk.md0p1.out‚ 572.4140625 disk.sdb2.in‚ 3.49609375 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 1.404296875 disk.nvme1n1.in‚ 200722.9375 disk.sdb1.out‚0) proc.memory.percent‚18.49517822265625 disk.rbd2.out‚ 1.341796875! disk.nvme0n1.in‚ 328970.015625 network.recv‚ 1.831625e+06$ disk./.usageGB‚2975.073703765869 disk.rbd2.in‚0# disk.nvme0n1.out‚28835.28515625 disk.sda1.out‚0Þ!c«2Â- _wandbruntime‚5457  _runtime‚5457õþCÆÃÀ & lr-AdamW/pg1‚0.0006322957064813915 & lr-AdamW/pg2‚0.0006322957064813915 trainer/global_step‚4699 % _timestamp‚1.7884694042403874e+09 _runtime‚ 5457.42542004 _step‚381‹$ËÄÅ _step‚381 & lr-AdamW/pg1‚0.0006322957064813915 & lr-AdamW/pg2‚0.0006322957064813915 trainer/global_step‚4699 % _timestamp‚1.7884694042403874e+09 _runtime‚ 5457.42542004ý%à ±2Å- _wandbruntime‚5457  _runtime‚5457­ùÆÙO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚199665586353152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9480037379072 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33747730 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚181237059654448 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88812187 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9320814581760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 127444727 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127471071 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9321048250368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚674 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _runtime‚5457.426210204 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 216283260 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86734420 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚211150167496192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52986018 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88812189 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9321048250368 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9294632801324 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚190344520099840 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33748402 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚190344538102784 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 trainer/global_step‚4699 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88812189 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚199665334681600 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ICDeviceStatsMonitor.on_train_batch_start/o¹½«ßæversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9480321707008 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚190531692455772 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚181237042615088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚190344520099840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚190531446435328 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚201670154057216 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88812187 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 216256914 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127471071 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199665334681600 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚190344538102784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86733746 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9294403820240 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104·µä'ÇÞO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88812187 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9320814581760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 127444727 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127471071 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9321048250368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚674 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _runtime‚5457.426210204 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 216283260 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86734420 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚211150167496192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52986018 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88812189 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9321048250368 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9294632801324 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚190344520099840 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33748402 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚190344538102784 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 trainer/global_step‚4699 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88812189 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚199665334681600 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9480321707008 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚190531692455772 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚181237042615088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚190344520099840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚190531446435328 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚201670154057216 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88812187 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 216256914 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127471071 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199665334681600 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚190344538102784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86733746 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9294403820240 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚199665586353152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9480037379072 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33747730 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚181237059654448 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777þye#2È- _wandbruntime‚5458  _runtime‚5458Œš¤é&ÉãM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 127497789 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 216301881 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 trainer/global_step‚4699 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9322778607616 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9323012289536 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 127497789 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚201711814087168 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚181274572972512 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88830464 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9322778607616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52996818 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9482321484288 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚190570935281738 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚199706836429312 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127471419 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚190383824139776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚199706584744448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9323012289536 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚211193849159680 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚190383806136832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚669 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52996820 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚667 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚211194159511552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 $ _timestamp‚1.788469405198702e+09 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127471419 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9482035072512 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 216301881 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216328253 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚201711838027264 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86751896 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 216328253 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88830462 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33755076 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚181274590011872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86751227 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚190383806136832 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88830464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚190571181302390 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9296362309226 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88830462 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _step‚383 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33754409 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096«Ó¦øÊèM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127471419 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9482035072512 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 216301881 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216328253 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚201711838027264 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86751896 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 216328253 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88830462 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33755076 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚181274590011872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86751227 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚190383806136832 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88830464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚190571181302390 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9296362309226 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88830462 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _step‚383 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33754409 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 127497789 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 216301881 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 trainer/global_step‚4699 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9322778607616 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9323012289536 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 127497789 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚201711814087168 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚181274572972512 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88830464 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9322778607616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52996818 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9482321484288 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚190570935281738 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚199706836429312 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127471419 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚190383824139776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚199706584744448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9323012289536 F8DeviceStatsMonitor.ÜÔ<Êöon_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚211193849159680 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚190383806136832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚669 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52996820 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚667 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚211194159511552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 $ _timestamp‚1.788469405198702e+09 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚679 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚201728103909376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87621489 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53526116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 128782679 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89725939 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 128782679 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚192506148367480 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9392341860748 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚213334254159360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚192308839952384 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 _step‚386 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89725937 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚201728103909376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9580214062080 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚192308821949440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34094694 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 128809023 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9579925539840 trainer/global_step‚4749 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚192308839952384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89725939 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9419030288384 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 % _timestamp‚1.7884694594013906e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚201727852237824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 218534962 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚213334566621696 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚183113560486288 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _runtime‚5512.586411858 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9419263956992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚677 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87620810 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚203754328619520 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34095371 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚183113577525648 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚192308821949440 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚192505902347036 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0;]¼å'ÛßO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚201728103909376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9580214062080 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚192308821949440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34094694 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 128809023 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9579925539840 trainer/global_step‚4749 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚192308839952384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89725939 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9419030288384 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 % _timestamp‚1.7884694594013906e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚201727852237824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 218534962 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚213334566621696 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚183113560486288 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _runtime‚5512.586411858 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9419263956992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚677 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87620810 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚203754328619520 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34095371 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚183113577525648 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚192308821949440 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚192505902347036 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 218508616 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 128809023 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚679 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚201728103909376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87621489 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53526116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 128782679 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89725939 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 128782679 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚192506148367480 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9392341860748 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚213334254159360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚192308839952384 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 _step‚386 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89725937 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136‚]žö.2Ü- _wandbruntime‚5513  _runtime‚5513<äS›ÝäM _runtime‚5513.598025173 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚213377930740224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9394529331026 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 128835741 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚203796012589568 trainer/global_step‚4749 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚192348125989376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚213378245286400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚183151107883072 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89744214 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53536918 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89744212 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9421227996160 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89744212 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9581918150656 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚201769102300672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚203796036529664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9420994314240 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87639363 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9582208756736 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚192348125989376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53536920 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9394300349734 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87638689 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 128809371 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34102443 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 % _timestamp‚1.7884694604126763e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚192348107986432 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚192545637214098 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚201769353985536 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚192348107986432 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 128835741 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9421227996160 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89744214 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9420994314240 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218579955 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 218553583 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 128809371 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34101771 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0€Jèï&ÞéM REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34101771 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5513.598025173 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚213377930740224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9394529331026 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 128835741 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚203796012589568 trainer/global_step‚4749 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚192348125989376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚213378245286400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚183151107883072 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89744214 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53536918 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89744212 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9421227996160 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89744212 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9581918150656 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚201769102300672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚203796036529664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9420994314240 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87639363 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9582208756736 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚192348125989376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53536920 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9394300349734 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87638689 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 128809371 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34102443 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 % _timestamp‚1.7884694604126763e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚192348107986432 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚192545637214098 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚201769353985536 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚192348107986432 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 128835741 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9421227996160 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89744214 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9420994314240 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218579955 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 218553583 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 128809371ƒ)éod2ß- _wandbruntime‚5513  _runtime‚5513ž$[àÕ 1loss/FBt1/assignment_loss‚0.13549725429293796 1loss/FBt4/assignment_loss‚0.07200816477292922 .loss/FRt3/detection_loss‚1.549969081977202 ,loss/FRt4/detection_loss‚2.7208754979074 epoch‚3 0loss/FRt3/assignment_loss‚1.5816395753814327 2loss/SRqqt1/assignment_loss‚1.5054136303233656 %loss/total_loss‚41.26812554333343 trainer/global_step‚4749 2loss/SRqqt4/assignment_loss‚1.2956956520210952 1loss/FBt2/assignment_loss‚0.17238946851973327 0loss/SRqqt1/detection_loss‚1.962768497302177 /loss/FBt2/detection_loss‚1.9993089416197367 2loss/SRqqt2/assignment_loss‚1.8422691092408936 2loss/SRqqt3/assignment_loss‚0.8989815814071335 1loss/SRqqt3/detection_loss‚1.0023957007098943 $ _timestamp‚1.788469460414972e+09 _runtime‚5513.599673181 0loss/FRt1/assignment_loss‚2.4124784554753984 /loss/FRt2/assignment_loss‚3.199948443018872 /loss/FRt1/detection_loss‚2.9869999310800006 /loss/FBt3/detection_loss‚0.5864306403402711 /loss/FBt4/detection_loss‚2.3123494436343512 1loss/FBt3/assignment_loss‚0.04523565954689371 .loss/FRt2/detection_loss‚2.505733443643017 1loss/SRqqt2/detection_loss‚2.2683463466578515 /loss/FBt1/detection_loss‚2.5736482089216057 0loss/FRt4/assignment_loss‚3.1822316851466894 1loss/SRqqt4/detection_loss‚2.4555111303925514 _step‚388!ÈP`áÚ 1loss/FBt3/assignment_loss‚0.04523565954689371 .loss/FRt2/detection_loss‚2.505733443643017 1loss/SRqqt2/detection_loss‚2.2683463466578515 /loss/FBt1/detection_loss‚2.5736482089216057 0loss/FRt4/assignment_loss‚3.1822316851466894 1loss/SRqqt4/detection_loss‚2.4555111303925514 _step‚388 1loss/FBt1/assignment_loss‚0.13549725429293796 1loss/FBt4/assignment_loss‚0.07200816477292922 .loss/FRt3/detection_loss‚1.549969081977202 ,loss/FRt4/detection_loss‚2.7208754979074 epoch‚3 0loss/FRt3/assignment_loss‚1.5816395753814327 2loss/SRqqt1/assignment_loss‚1.5054136303233656 %loss/total_loss‚41.26812554333343 trainer/global_step‚4749 2loss/SRqqt4/assignment_loss‚1.2956956520210952 1loss/FBt2/assignment_loss‚0.17238946851973327 0loss/SRqqt1/detection_loss‚1.962768497302177 /loss/FBt2/detection_loss‚1.9993089416197367 2loss/SRqqt2/assignment_loss‚1.8422691092408936 2loss/SRqqt3/assignment_loss‚0.8989815814071335 1loss/SRqqt3/detection_loss‚1.0023957007098943 $ _timestamp‚1.788469460414972e+09 _runtime‚5513.599673181 0loss/FRt1/assignment_loss‚2.4124784554753984 /loss/FRt2/assignment_loss‚3.199948443018872 /loss/FRt1/detection_loss‚2.9869999310800006 /loss/FBt3/detection_loss‚0.5864306403402711 /loss/FBt4/detection_loss‚2.3123494436343512„)’Sâ‚(:… ÚÁçÔÐÝ㇠gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚116.916' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.229 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08¸Óhrã‚(:ç ÚÁçÔ÷ñºˆ proc.memory.rssMB‚3030.25 cpu‚25.13611998813824 network.recv‚ 1.855258e+06) disk./.usagePercent‚83.14641722152898# disk.nvme0n1.out‚29754.44140625 disk.md0p1.in‚7.125 disk.rbd2.out‚ 1.451171875$ memory_percent‚34.98954772949219) proc.memory.percent‚18.49517822265625 disk.sdb2.in‚ 3.49609375 disk.sdb2.out‚601.9716796875 disk.rbd0.in‚0.015625& proc.memory.availableMB‚ 10651.3125 disk.md0p1.out‚ 580.58203125 disk.md0.in‚7.125" disk.nvme1n1.out‚ 40857.6640625 network.sent‚ 1.1196155e+07 disk.dm-0.in‚529692.95703125 disk.rbd0.out‚ 1.5576171875 disk.sdb1.out‚0 disk.sda.in‚ 3.62890625! disk.nvme0n1.in‚ 328970.015625# disk.nvme1n1.in‚200722.94140625 disk.dm-0.out‚70612.10546875 disk.sda2.in‚ 3.62890625 disk.sda2.out‚601.9716796875 disk.md0.out‚ 580.58203125 disk.sda.out‚601.9716796875 disk.sda1.in‚0 disk.rbd2.in‚0 disk.sdb.out‚601.9716796875 proc.cpu.threads‚17$ disk./.usageGB‚2973.718620300293 disk.sda1.out‚0 disk.sdb.in‚ 3.49609375 disk.sdb1.in‚0Rà|ä‚(:ñ éÁçÔ÷˜„‡ network.sent‚ 1.130337e+07 disk.md0.in‚7.125! disk.nvme0n1.in‚ 328970.015625 proc.cpu.threads‚17! disk.sdb2.out‚603.38525390625# disk.nvme1n1.in‚200722.94140625$ memory_percent‚34.98837947845459 disk.sdb2.in‚ 3.49609375 disk.rbd0.out‚ 1.5576171875 disk.rbd2.out‚ 1.4951171875# disk.nvme1n1.out‚40901.75390625 disk.dm-0.in‚529692.95703125 disk.sda2.in‚ 3.62890625 disk.md0p1.out‚ 581.94140625 disk.sdb.out‚603.38525390625 disk.sdb1.out‚0 proc.memory.rssMB‚3030.25) proc.memory.percent‚18.49517822265625) disk./.usagePercent‚83.15474111611628 disk.rbd0.in‚0.015625 disk.sda1.in‚0 disk.md0p1.in‚7.125 disk.rbd2.in‚0 disk.sdb1.in‚0 disk.sda.in‚ 3.62890625 network.recv‚ 1.862048e+06 disk.dm-0.out‚70768.21484375 disk.sda1.out‚0! disk.sda2.out‚603.38525390625 disk.md0.out‚ 581.94140625" disk.nvme0n1.out‚ 29866.4609375 cpu‚25.135811788433244% disk./.usageGB‚2974.0163230895996 disk.sdb.in‚ 3.49609375 disk.sda.out‚603.38525390625* proc.memory.availableMB‚10651.50390625ÞÓ/’å‚(:‡ éÁçÔÿœÁ‡ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚116.583' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.14575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00æ‡z‘æ‚(:† øÁçÔ…Å߇ gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚138.642' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.6605 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0² öB_ç‚(:Ô øÁçÔ»çü‡% disk./.usageGB‚2974.0163078308105 disk.md0p1.in‚7.125 disk.md0p1.out‚ 583.453125 disk.rbd0.out‚ 1.62060546875 network.sent‚ 1.1311575e+07# disk.nvme1n1.in‚200722.94140625 disk.sdb.in‚ 3.49609375 disk.sdb.out‚ 604.953125 disk.sda.out‚ 604.953125$ proc.memory.availableMB‚10651.25 proc.memory.rssMB‚3030.25 disk.rbd2.in‚0) disk./.usagePercent‚83.15474068947417 disk.rbd0.in‚0.015625! disk.nvme0n1.in‚ 328970.015625# disk.nvme0n1.out‚29868.55078125 cpu‚25.135482837846766 disk.sdb2.in‚ 3.49609375 disk.sdb1.out‚0 disk.dm-0.in‚529692.95703125 network.recv‚ 1.866181e+06 disk.md0.out‚ 583.453125 disk.sdb1.in‚0 disk.sdb2.out‚ 604.953125# disk.nvme1n1.out‚41031.08984375 disk.sda1.out‚0$ memory_percent‚34.98992919921875 disk.dm-0.out‚ 70899.640625 disk.sda1.in‚0 disk.sda2.in‚ 3.62890625 disk.sda2.out‚ 604.953125 proc.cpu.threads‚17 disk.md0.in‚7.125 disk.rbd2.out‚ 1.4951171875 disk.sda.in‚ 3.62890625) proc.memory.percent‚18.49517822265625»i2U2è- _wandbruntime‚5564  _runtime‚5564M_C ÇéÁ & lr-AdamW/pg1‚0.0006221414164568839 & lr-AdamW/pg2‚0.0006221414164568839 trainer/global_step‚4799 % _timestamp‚1.7884695113718808e+09 _runtime‚5564.557172589 _step‚389î–DæÌêÆ _step‚389 & lr-AdamW/pg1‚0.0006221414164568839 & lr-AdamW/pg2‚0.0006221414164568839 trainer/global_step‚4799 % _timestamp‚1.7884695113718808e+09 _runtime‚5564.557172589…Ï«2ë- _wandbruntime‚5564  _runtime‚5564Àü`nìÚO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 220786664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90639689 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚194273123799040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88509961 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚203790621465600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54066218 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚215518289759744 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9680049062912 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884695113732858e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34443743 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90639687 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚184990078357488 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚194273123799040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54066216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9490279901256 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚203790369794048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚194273141801984 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚205838527121920 ^LDeviceStatsMonitor.on_tr@ýð‘rain_batch_start/requested_bytes.small_pool.allocated‚ 9490508882340 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚5564.558923187 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9679762637824 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _step‚390 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9517479663616 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 220786664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90639689 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚194273123799040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88509961 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚203790621465600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54066218 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚215518289759744 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9680049062912 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884695113732858e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34443743 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90639687 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚184990078357488 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚194273123799040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54066216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9490279901256 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚203790369794048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚194273141801984 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚205838527121920 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9490508882340 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚5564.558923187 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9679762637824 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _step‚390 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9517479663616 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 J-‘î‚(:† ‡ÂçÔ–ù„‰ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚94.857' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚23.71425 gpu.0.smClock‚1200 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Rþ´¥cï‚(:Ø ‡ÂçÔ¹ìÁ‰% disk./.usageGB‚2974.0146713256836 disk.dm-0.out‚ 71001.4296875 disk.md0.out‚584.8125" disk.nvme1n1.out‚ 41132.7734375 disk.nvme0n1.in‚ 328970.03125 disk.rbd0.in‚0.015625 proc.cpu.threads‚17 network.sent‚ 1.1327543e+07 disk.sda1.out‚0 disk.sdb2.in‚ 3.49609375$ memory_percent‚34.98983383178711 disk.nvme0n1.out‚ 29868.65625 disk.sdb2.out‚ 606.36328125 disk.sdb.in‚ 3.49609375 proc.memory.rssMB‚3030.25 network.recv‚ 1.870081e+06 disk.dm-0.in‚529692.97265625 disk.sda2.in‚ 3.62890625 disk.md0p1.out‚584.8125 disk.rbd2.out‚ 1.55810546875# disk.nvme1n1.in‚200722.94140625( proc.memory.availableMB‚ 10651.265625 disk.sdb.out‚ 606.36328125 disk.sdb1.out‚0) disk./.usagePercent‚83.15469493210588 disk.sda1.in‚0 disk.md0p1.in‚7.125 disk.rbd0.out‚ 1.62060546875 disk.sda.in‚ 3.62890625 disk.sda.out‚ 606.36328125 disk.rbd2.in‚0 disk.sdb1.in‚0) proc.memory.percent‚18.49517822265625 cpu‚25.13514995968055 disk.sda2.out‚ 606.36328125 disk.md0.in‚7.125Ó ~2ð- _wandbruntime‚5565  _runtime‚5565*?ué&ñãM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130173693 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 $ _timestamp‚1.788469512483626e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220831657 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9681771836416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130173693 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚203831871541760 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚203831871541760 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚203831619856896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130147323 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88527393 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9519443702784 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9492238390242 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220805285 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚205880235032064 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88526714 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9682056151040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 220831657 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚194312427838976 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚185027608714912 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90657962 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54077018 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90657962 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚677 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚194312409836032 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34449696 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚185027625754272 _runtime‚5565.669694203 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9519443702784 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9492238390242 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220805285 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚205880235032064 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88526714 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9682056151040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 220831657 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚194312427838976 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚185027608714912 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90657962 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54077018 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90657962 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚677 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚194312409836032 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34449696 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚185027625754272 _runtime‚5565.669694203 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220831657 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9681771836416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130173693 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚203831871541760 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚203831871541760 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚203831619856896‡|¤“42ó- _wandbruntime‚5565  _runtime‚5565¬,ZôÔ 0loss/FRt2/assignment_loss‚3.1147839228312173 0loss/FRt3/assignment_loss‚1.2844312515175134 1loss/FBt2/assignment_loss‚0.04747368477377346 .loss/FRt2/detection_loss‚2.073351197452165 .loss/FBt1/detection_loss‚3.008586117199489 /loss/FRt1/assignment_loss‚3.157754413286845 1loss/FBt1/assignment_loss‚0.11968060332570271 /loss/FBt2/detection_loss‚2.2365101359107276 /loss/FBt4/detection_loss‚2.6674852550029753 1loss/FBt3/assignment_loss‚0.07398540152642141 /loss/SRqqt2/detection_loss‚2.49422166368057 %loss/total_loss‚44.10166091566994 2loss/SRqqt4/assignment_loss‚1.7740588368791523 0loss/FBt4/assignment_loss‚0.3222227290704723 /loss/FRt3/detection_loss‚1.7485336052221165 1loss/SRqqt1/detection_loss‚1.8893857157001128 epoch‚3 2loss/SRqqt1/assignment_loss‚1.3106866357131646 1loss/SRqqt2/assignment_loss‚1.713691133941556 .loss/FRt4/detection_loss‚4.008287247489481 trainer/global_step‚4799 % _timestamp‚1.7884695124858801e+09 2loss/SRqqt3/assignment_loss‚1.1811770868077074 /loss/FRt1/detection_loss‚2.6137513478597003 1loss/SRqqt3/detection_loss‚0.8090636501869848 1loss/SRqqt4/detection_loss‚2.4320153070218637 /loss/FBt3/detection_loss‚0.5990255939836303 _step‚392 /loss/FRt4/assignment_loss‚3.421498379286598 _runtime‚5565.670710068Çð_õÙ /loss/FBt4/detection_loss‚2.6674852550029753 1loss/FBt3/assignment_loss‚0.07398540152642141 /loss/SRqqt2/detection_loss‚2.49422166368057 %loss/total_loss‚44.10166091566994 2loss/SRqqt4/assignment_loss‚1.7740588368791523 0loss/FBt4/assignment_loss‚0.3222227290704723 /loss/FRt3/detection_loss‚1.7485336052221165 1loss/SRqqt1/detection_loss‚1.8893857157001128 epoch‚3 2loss/SRqqt1/assignment_loss‚1.3106866357131646 1loss/SRqqt2/assignment_loss‚1.713691133941556 .loss/FRt4/detection_loss‚4.008287247489481 trainer/global_step‚4799 % _timestamp‚1.7884695124858801e+09 2loss/SRqqt3/assignment_loss‚1.1811770868077074 /loss/FRt1/detection_loss‚2.6137513478597003 1loss/SRqqt3/detection_loss‚0.8090636501869848 1loss/SRqqt4/detection_loss‚2.4320153070218637 /loss/FBt3/detection_loss‚0.5990255939836303 _step‚392 /loss/FRt4/assignment_loss‚3.421498379286598 _runtime‚5565.670710068 0loss/FRt2/assignment_loss‚3.1147839228312173 0loss/FRt3/assignment_loss‚1.2844312515175134 1loss/FBt2/assignment_loss‚0.04747368477377346 .loss/FRt2/detection_loss‚2.073351197452165 .loss/FBt1/detection_loss‚3.008586117199489 /loss/FRt1/assignment_loss‚3.157754413286845 1loss/FBt1/assignment_loss‚0.11968060332570271 /loss/FBt2/detection_loss‚2.2365101359107276ˆé¸wmö‚(:â –ÂçÔ™Ú‡ disk.rbd0.out‚ 1.68359375 network.sent‚ 1.1411123e+07 network.recv‚ 1.876837e+06" disk.nvme0n1.out‚ 29870.8046875 disk.dm-0.in‚529692.97265625# memory_percent‚34.9902868270874 cpu‚25.134854763511274 disk.sdb.in‚ 3.49609375 disk.sda2.out‚ 608.466796875 disk.md0.in‚7.125* proc.memory.availableMB‚10651.19140625 disk.md0.out‚ 586.84765625 disk.sdb1.out‚0 disk.sda.in‚ 3.62890625 disk.sda1.in‚0 disk.sda1.out‚0 proc.memory.rssMB‚3030.25 proc.cpu.threads‚17 disk.sdb2.out‚ 608.466796875 disk.rbd2.out‚ 1.55810546875# disk.nvme1n1.in‚200722.94140625 disk.sda.out‚ 608.466796875) disk./.usagePercent‚83.14378153311972 disk.dm-0.out‚ 71113.2109375) proc.memory.percent‚18.49517822265625$ disk./.usageGB‚2973.624355316162 disk.md0p1.in‚7.125 disk.sdb2.in‚ 3.49609375 disk.rbd2.in‚0 disk.nvme1n1.out‚ 41242.40625 disk.nvme0n1.in‚ 328970.03125 disk.sda2.in‚ 3.62890625 disk.md0p1.out‚ 586.84765625 disk.rbd0.in‚0.015625 disk.sdb.out‚ 608.466796875 disk.sdb1.in‚0ps¡÷‚(:… –ÂçÔ«ù£‡ gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚144.212' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.053 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jaºp‘ø‚(:† ¥ÂçÔ­ãŇ gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚89.287' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚22.32175 gpu.0.smClock‚1170 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»b‡clù‚(:á ¥ÂçÔÛëî‡# disk.nvme1n1.out‚41371.80859375 disk.sdb1.out‚0 disk.sda.out‚609.9443359375 disk.nvme0n1.in‚ 328970.03125 disk.md0p1.out‚ 588.265625 disk.sdb2.in‚ 3.49609375 disk.md0.in‚7.125 disk.sdb2.out‚609.9443359375$ memory_percent‚34.99100208282471) disk./.usagePercent‚83.14724768043207 disk.sdb.out‚609.9443359375 disk.sda2.in‚ 3.62890625 disk.sda2.out‚609.9443359375 disk.md0p1.in‚7.125 network.recv‚ 1.883399e+06$ disk./.usageGB‚2973.748321533203 disk.md0.out‚ 588.265625 disk.rbd2.out‚ 1.62109375 disk.rbd0.out‚ 1.68359375# disk.nvme1n1.in‚200722.94140625 disk.sdb.in‚ 3.49609375 disk.sda.in‚ 3.62890625 cpu‚25.1345338522758 network.sent‚ 1.1507973e+07 disk.rbd0.in‚0.015625 disk.sda1.out‚0 disk.rbd2.in‚0 disk.sdb1.in‚0# disk.nvme0n1.out‚29870.86328125* proc.memory.availableMB‚10651.07421875 proc.memory.rssMB‚3030.25) proc.memory.percent‚18.49517822265625 disk.dm-0.in‚529692.97265625 disk.dm-0.out‚ 71242.671875 disk.sda1.in‚0 proc.cpu.threads‚17[kú‚(:… ´ÂçÔ®òÛ‡ gpu.0.gpu‚39.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚139.868' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.967 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€†•sû‚(:è ´ÂçÔá爈 disk.sdb.out‚611.31787109375 disk.nvme0n1.in‚ 328970.03125 disk.sda1.out‚0! disk.sda2.out‚611.31787109375 disk.md0.out‚ 589.5703125$ memory_percent‚34.98945236206055$ disk./.usageGB‚2973.822410583496 disk.sdb1.in‚0 disk.sda.in‚ 3.62890625 disk.dm-0.in‚529692.97265625 disk.dm-0.out‚71402.94921875 disk.md0p1.in‚7.125 disk.md0p1.out‚ 589.5703125! disk.sdb2.out‚611.31787109375# disk.nvme1n1.in‚200722.94140625# disk.nvme1n1.out‚41529.09765625 disk.sda.out‚611.31787109375 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 1.68359375 disk.sdb.in‚ 3.49609375 disk.md0.in‚7.125 network.sent‚ 1.1516116e+07 disk.rbd2.in‚0 disk.rbd2.out‚ 1.62109375 disk.sda2.in‚ 3.62890625( proc.memory.availableMB‚ 10651.328125 proc.memory.rssMB‚3030.25 proc.cpu.threads‚17 network.recv‚ 1.887433e+06 disk.sdb2.in‚ 3.49609375 cpu‚25.138515732654586 disk.sdb1.out‚0" disk.nvme0n1.out‚ 29873.8515625) disk./.usagePercent‚83.14931924128695 disk.sda1.in‚0) proc.memory.percent‚18.49517822265625ÜúÑá‘ü‚(:† ÃÂçÔ÷Ї gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K"$rý‚(:ç ÃÂçÔÚ­ƒˆ network.sent‚ 1.1521487e+07% disk./.usageGB‚2974.0721549987793 disk.sda.in‚ 3.62890625 disk.sdb.in‚ 3.49609375$ memory_percent‚34.98642444610596) proc.memory.percent‚18.48917007446289) disk./.usagePercent‚83.15630219966442 disk.sdb1.out‚0 disk.dm-0.in‚529692.97265625 disk.dm-0.out‚71546.71484375 disk.md0.in‚7.125 disk.sdb1.in‚0 disk.nvme0n1.in‚ 328970.03125 disk.sda1.in‚0 disk.rbd2.out‚ 1.62109375 disk.sda1.out‚0 disk.md0.out‚ 591.46484375# disk.nvme0n1.out‚29873.91796875 disk.sda2.in‚ 3.62890625 disk.md0p1.out‚ 591.46484375 disk.sdb2.out‚ 613.267578125 disk.rbd0.out‚ 1.68359375! disk.nvme1n1.out‚ 41672.796875! proc.memory.rssMB‚ 3029.265625 proc.cpu.threads‚17 disk.md0p1.in‚7.125 disk.rbd2.in‚0 disk.sdb.out‚ 613.267578125* proc.memory.availableMB‚10651.82421875 network.recv‚ 1.891097e+06 disk.sda.out‚ 613.267578125 disk.sda2.out‚ 613.267578125 disk.sdb2.in‚ 3.49609375 disk.rbd0.in‚0.015625# disk.nvme1n1.in‚200722.94140625 cpu‚25.14919447924739—Òîwþ‚(:ì ÒÂçÔݱ±ˆ disk.rbd0.in‚0.015625 network.recv‚ 1.894549e+06% disk./.usageGB‚2974.5737113952637 disk.sdb.out‚614.87255859375 disk.sdb1.in‚0 disk.md0p1.in‚7.125 disk.rbd2.in‚0 disk.dm-0.out‚71671.25390625 disk.md0p1.out‚ 593.00390625! proc.memory.rssMB‚ 3029.265625) proc.memory.percent‚18.48917007446289 proc.cpu.threads‚17 disk.nvme0n1.out‚ 29874.0625 disk.sda1.in‚0! disk.sdb2.out‚614.87255859375 disk.rbd2.out‚ 1.62109375# disk.nvme1n1.out‚41797.19140625 disk.sdb.in‚ 3.49609375 disk.rbd0.out‚ 1.68359375 disk.sda.in‚ 3.62890625 disk.sdb1.out‚0! disk.sda2.out‚614.87255859375 disk.sda.out‚614.87255859375 disk.nvme0n1.in‚ 328970.03125 disk.md0.in‚7.125$ memory_percent‚34.98430252075195 cpu‚25.160033456852666 network.sent‚ 1.1526856e+07) disk./.usagePercent‚83.17032592642778# disk.nvme1n1.in‚200722.94140625 disk.dm-0.in‚529692.97265625 disk.sda1.out‚0 disk.sda2.in‚ 3.62890625 disk.md0.out‚ 593.00390625( proc.memory.availableMB‚ 10652.171875 disk.sdb2.in‚ 3.49609375ìöâ=’ÿ‚(:‡ ÒÂçÔ樿‡ gpu.0.gpu‚42.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚141.087' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.27175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“»j²‘€‚(:† áÂçÔðö±† gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0PµÓ1u‚(:ê áÂçÔþÞч network.recv‚ 1.898884e+06 disk.rbd0.out‚ 1.68359375# disk.nvme0n1.out‚29874.12109375 disk.md0p1.in‚7.125 disk.sdb2.out‚616.3798828125 disk.md0.out‚ 594.44921875 disk.rbd0.in‚0.015625 disk.dm-0.out‚ 72064.3359375 disk.md0p1.out‚ 594.44921875 disk.rbd2.in‚0 disk.sdb.out‚616.3798828125 disk.sdb1.in‚0 proc.memory.rssMB‚ 3028.28125 cpu‚25.17122095171446 disk.rbd2.out‚ 1.62109375 disk.sda.out‚616.3798828125 disk.sda2.in‚ 3.62890625) proc.memory.percent‚18.48316192626953 proc.cpu.threads‚17( disk./.usagePercent‚83.1702841154992# disk.nvme1n1.out‚42190.21484375 disk.sdb1.out‚0 network.sent‚ 1.1530499e+07 disk.sda.in‚ 3.62890625 disk.nvme0n1.in‚ 328970.03125 disk.sda1.in‚0% disk./.usageGB‚2974.5722160339355 disk.sdb2.in‚ 3.49609375 disk.md0.in‚7.125# disk.nvme1n1.in‚200722.94140625* proc.memory.availableMB‚10652.91796875 disk.sdb.in‚ 3.49609375 disk.dm-0.in‚529692.97265625 disk.sda1.out‚0 disk.sda2.out‚616.3798828125$ memory_percent‚34.97974872589111»UÙ­2‚- _wandbruntime‚5663  _runtime‚5663mY?ëE*ƒ¿T `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 9613390155776 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚218645632725504 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚208868559940096 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 9585962686750 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 7134511104 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚92069582 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 175812096 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚206813139549184 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 9613390155776 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚92069580 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 131572147 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚206813139549184 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6966900736 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 161829376 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20711 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 9613214343680 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚92069582 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3532 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚197217087100894 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚55038308 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚197199943208448 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 560288777 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20713 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚218645818494976 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 347418624 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚197217275960726 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 7134511104 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚582 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚385 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚309 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 185769472 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 223641729 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 223621016 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 131572147 % _timestamp‚1.7884696101599643e+09 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚206813333364224 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 131551436 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚34976305 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚206813139549184 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 9613390155776 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚92069580 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 131572147 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚206813139549184 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6966900736 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 161829376 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20711 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 9613214343680 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚92069582 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3532 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚197217087100894 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚55038308 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚197199943208448 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 560288777 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20713 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚218645818494976 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 347418624 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚197217275960726 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 7134511104 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚582 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚385 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚309 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 185769472 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 223641729 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 223621016 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 131572147 % _timestamp‚1.7884696101599643e+09 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚206813333364224 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 131551436 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚34976305 FDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚273 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3532 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚34981869 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚273 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20711 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 560288777 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚309 % _timestamp‚1.7884696112315338e+09 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 9777546645504 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚582 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚206843872898560 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 9613864015872 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚309 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 223666019 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 175812096 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 347418624 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188859832 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚309 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 185769472 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 171820472 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 572522496 _step‚394 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3532 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚197229990879744 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 223666019 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚206843872898560 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 9613864015872 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6966900736 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6741059457 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6966900736 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚273 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 131581055 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 9586606715034 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 9613688203776 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚92084964 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚90031473 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1319 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚187656701103136 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚197229990879744 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚187656684063776 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 9777708474880 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚582 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 193815040 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚206843679083520 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚55049992 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚386 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚197243118958338 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚92084962 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚388 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚208898945633792 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 131560344 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚92084964 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 6236507136 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚36 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚197230008882688 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 223645306 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 131560344 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 131581055 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 9613688203776 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0¯sžV)‡ÐR XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚197243118958338 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚92084962 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚388 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚208898945633792 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 131560344 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚92084964 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 6236507136 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚36 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚197230008882688 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 223645306 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 131560344 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 131581055 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 9613688203776 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 193815040 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚34981483 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 223645306 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚197230008882688 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚90031861 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1300 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 161829376 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚582 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 175812096 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚55049990 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚197243307818170 _runtime‚5664.417075174 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚92084962 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 565025792 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20713 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚273 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3532 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚34981869 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚273 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20711 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 560288777 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚309 % _timestamp‚1.7884696112315338e+09 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 9777546645504 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚582 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚206843872898560 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 9613864015872 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚309 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 223666019 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 175812096 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 347418624 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188859832 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚309 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 185769472 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 171820472 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 572522496 _step‚394 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3532 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚197229990879744 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 223666019 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚206843872898560 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 9613864015872 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6966900736 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6741059457 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6966900736 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚273 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 131581055 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 9586606715034 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 9613688203776 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚92084964 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚90031473 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1319 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚187656701103136 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚197229990879744 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚187656684063776 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 9777708474880 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚582 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 193815040 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚206843679083520 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚55049992 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚386ŠünÍœˆ‚(:‘ ðÂçÔ± ¦† gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ[ÔOu‰‚(:ê ðÂçÔæ万) disk./.usagePercent‚83.17663639015024$ disk./.usageGB‚2974.799404144287! disk.sdb2.out‚618.17724609375 disk.rbd0.in‚0.015625 disk.sdb1.in‚0 disk.sda1.in‚0! disk.sda2.out‚618.17724609375 disk.sdb.in‚ 3.49609375 disk.sda1.out‚0 disk.md0p1.out‚ 596.18359375 disk.rbd0.out‚ 1.68359375 disk.dm-0.in‚529692.97265625 disk.sda2.in‚ 3.62890625 proc.memory.rssMB‚ 3028.28125 disk.md0p1.in‚7.125 disk.md0.in‚7.125 disk.sdb1.out‚0 disk.sda.out‚618.17724609375 disk.nvme0n1.out‚ 29874.4375% memory_percent‚34.978389739990234) proc.memory.percent‚18.48316192626953 network.sent‚ 1.1536681e+07 disk.rbd2.out‚ 1.62109375# disk.nvme1n1.out‚42409.51171875 disk.sda.in‚ 3.62890625 proc.cpu.threads‚17 network.recv‚ 1.90338e+06 disk.nvme0n1.in‚ 328970.03125 disk.dm-0.out‚72283.94921875 cpu‚25.182607618286728# disk.nvme1n1.in‚200722.94140625 disk.sdb.out‚618.17724609375 disk.sdb2.in‚ 3.49609375 disk.md0.out‚ 596.18359375 disk.rbd2.in‚0( proc.memory.availableMB‚ 10653.140625ZÃ?=2Š- _wandbruntime‚5679  _runtime‚5679†®î;ç‹‚¥ 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9952331185340881 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 _runtime‚5679.694579905 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14015352725982666 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7658209204673767 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 _step‚395 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2605401575565338 jet/accuracy_5_of_7‚0 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5055035352706909 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3104563355445862 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.18008598685264587 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16252003610134125 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5652270913124084 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9992784857749939 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.44736841320991516 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5589508414268494 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.17045454680919647 *jet/accuracy_1_of_1‚0.4991779029369354 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4628470242023468 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.46081089973449707 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6754385828971863 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2202380895614624 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0416666679084301 particle/accuracy_5_of_6‚1 *"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.5 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9881494641304016 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9938775897026062 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974977374076843 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 particle/accuracy_2_of_6‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.523150622844696 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4642857015132904 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.38418903946876526 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08703155070543289 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.2a-#¢}9629626870155334 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_7‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.12874630093574524 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07434423267841339 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_7‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5229007005691528 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5198968648910522 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02234465815126896 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.48245614767074585 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5962302088737488 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5948275923728943 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.027188941836357117 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ,jet/accuracy_3_of_6‚0.057692307978868484 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.326299786567688 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9917161464691162 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1493576020002365 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9722222685813904 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.08506228029727936 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_7_of_7‚0 particle/accuracy_2_of_7‚1 7"Purity/*FRt1SRqqt0FBt/event_purity‚0.28408083319664 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5199999809265137 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2557367980480194 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41195863485336304 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.136943519115448 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1599142849445343 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.41470590233802795 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.14952129125595093 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_4_of_7‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2613767087459564 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1827145516872406 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4468390941619873 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.49108150601387024 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983749389648438 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.8888888955116272 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9906398057937622 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03231929615139961 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.38333332538604736 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 8"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.446368932723999 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01554129458963871 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 +jet/accuracy_2_of_4‚0.39119818806648254 jet/accuracy_1_of_6‚1 particle/accuracy_4_of_6‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.40920087695121765 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.43117815256118774 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988309144973755 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8611111044883728 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4722222089767456 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15986546874046326 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.00907808355987072 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14641985297203064 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5035097599029541 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1324712634086609 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.988300621509552 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4729063808917999 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.1158900111913681 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4554263651371002 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.5685932040214539 particle/accuracy_3_of_5‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8866235017776489 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.008620689623057842 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14029325544834137 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9875598549842834 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 particle/accuracy_2_of_2‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2595468759536743 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9892221689224243 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5771604776382446 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 +jet/accuracy_2_of_3‚0.24680721759796143 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05342261865735054 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.28611114621162415 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 5 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.38112872838974 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4889514148235321 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.565019428730011 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.16091953217983246 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.2777777910232544 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21266232430934906 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.3125 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5921769142150879 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1941024363040924 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03525061905384064 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.43407928943634033 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.12423178553581238 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.41655775904655457 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7857142686843872 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5392869114875793 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00570973614230752 jet/accuracy_5_of_5‚0 (particle/f_score‚0.40541210770606995 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.006134077440947294 *jet/accuracy_1_of_5‚0.9471837878227234 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 *jet/accuracy_2_of_6‚0.5769230723381042 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9919160604476929 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.36852678656578064 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 +particle/sensitivity‚0.3135445713996887 7validation_average_jet_accuracy‚0.44616997241973877 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9858116507530212 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1363026648759842 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.018834032118320465 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5292433500289917 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1300831437110901 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.533282995223999 particle/accuracy_1_of_3‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5369405150413513 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4512481391429901 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 +particle/specificity‚0.9569346308708191 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5110954642295837 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.23268677294254303 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1219298243522644 particle/accuracy_1_of_2‚1 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.34907400608062744 *"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9021164774894714 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6391556859016418 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4586028754711151 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4166666865348816 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09978359937667847 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13723443448543549 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20338910818099976 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.17420634627342224 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4552631974220276 particle/accuracy_3_of_6‚1 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6944704055786133 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9906432628631592 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4695286452770233 ,jet/accuracy_3_of_3‚0.027483582496643066 particle/accuracy_4_of_5‚1 jet/accuracy_2_of_7‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9866799712181091 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3192148506641388 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5118090510368347 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1382978856563568 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1666666716337204 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.15020577609539032 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1653771549463272 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5016317367553711 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.040566038340330124 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5743104219436646 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10087718814611435 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.12878789007663727 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5483297109603882 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ( Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5 +jet/accuracy_3_of_5‚0.13250769674777985 particle/accuracy_3_of_4‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1621096432209015 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5146566033363342 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0011337868636474013 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 epoch‚3 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3591424226760864 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1197916641831398 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.8871528506278992 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4799298346042633 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.35004323720932007 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5994054079055786 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2878500819206238 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 particle/accuracy_1_of_7‚1 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9803444147109985 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21469272673130035 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9479166865348816 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5389721393585205 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.210431307554245 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.015696648508310318 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5106180906295776 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6875 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 7"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.47858726978302 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5354989171028137 *jet/accuracy_1_of_4‚0.9208396077156067 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1844136118888855 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 (particle/accuracy‚0.8564659953117371 jet/accuracy_1_of_7‚0.5 particle/accuracy_3_of_7‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9904184341430664 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 +jet/accuracy_4_of_5‚0.01644265279173851 particle/accuracy_6_of_6‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5203709602355957 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1610453724861145 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5961804986000061 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 particle/accuracy_2_of_4‚1 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5392869114875793 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5464590191841125 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18959079682826996 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9823829531669617 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2639472186565399 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.18518085777759552 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1939181685447693 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.02054044045507908 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09790917485952377 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 5 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.36086305975914 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 :"Purity/1FRt2SRqqt*FBt/event_purity‚0.00186741363722831 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9944444298744202 particle/accuracy_5_of_5‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5543768405914307 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 particle/accuracy_1_of_4‚1 particle/accuracy_4_of_4‚1 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1968223601579666 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5347222089767456 jet/accuracy_4_of_6‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9867097735404968 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.16554054617881775 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0416666679084301 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.988183319568634 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *loss/val_total_loss‚41.833160325672296 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24965134263038635 particle/accuracy_1_of_5‚1 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04597701132297516 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5642856955528259 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4901565909385681 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.34340277314186096 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5659685134887695 *jet/accuracy_2_of_2‚0.1087983027100563 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9899565577507019 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 $ _timestamp‚1.788469626507405e+09 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.30000001192092896 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.524760365486145 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.35646504163742065 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8137417435646057 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5752976536750793 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4348784387111664 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5218191146850586 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07669685781002045 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 trainer/global_step‚4835 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4081568419933319 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.11777777969837189 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5380234122276306 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.28672876954078674 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5680246353149414 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5636757016181946 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5951150059700012 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5138888955116272 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 particle/accuracy_5_of_7‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5384024977684021 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9875683188438416 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1577407568693161 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3437499701976776 particle/accuracy_1_of_6‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9900316596031189 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5546320676803589 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1610453724861145 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3086419701576233 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14758478105068207 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.024611370638012886 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6875 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.43560609221458435 particle/accuracy_2_of_3‚1 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5285612344741821 6 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.270114928483963 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 particle/accuracy_6_of_7‚1 particle/accuracy_7_of_7‚0.5 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04463891685009003 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.12885595858097076 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ¥‰ªPŒ‡¥ 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6944704055786133 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9906432628631592 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4695286452770233 ,jet/accuracy_3_of_3‚0.027483582496643066 particle/accuracy_4_of_5‚1 jet/accuracy_2_of_7‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9866799712181091 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3192148506641388 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&PLNQ"ùurity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5118090510368347 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1382978856563568 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1666666716337204 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.15020577609539032 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1653771549463272 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5016317367553711 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.040566038340330124 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5743104219436646 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10087718814611435 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.12878789007663727 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5483297109603882 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ( Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5 +jet/accuracy_3_of_5‚0.13250769674777985 particle/accuracy_3_of_4‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1621096432209015 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5146566033363342 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0011337868636474013 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 epoch‚3 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3591424226760864 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1197916641831398 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.8871528506278992 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4799298346042633 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.35004323720932007 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5994054079055786 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2878500819206238 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 particle/accuracy_1_of_7‚1 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9803444147109985 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21469272673130035 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9479166865348816 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5389721393585205 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.210431307554245 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.015696648508310318 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5106180906295776 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6875 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 7"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.47858726978302 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5354989171028137 *jet/accuracy_1_of_4‚0.9208396077156067 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1844136118888855 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 (particle/accuracy‚0.8564659953117371 jet/accuracy_1_of_7‚0.5 particle/accuracy_3_of_7‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9904184341430664 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 +jet/accuracy_4_of_5‚0.01644265279173851 particle/accuracy_6_of_6‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5203709602355957 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1610453724861145 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5961804986000061 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 particle/accuracy_2_of_4‚1 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5392869114875793 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5464590191841125 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18959079682826996 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9823829531669617 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2639472186565399 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.18518085777759552 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1939181685447693 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.02054044045507908 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09790917485952377 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 5 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.36086305975914 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 :"Purity/1FRt2SRqqt*FBt/event_purity‚0.00186741363722831 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9944444298744202 particle/accuracy_5_of_5‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5543768405914307 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 particle/accuracy_1_of_4‚1 particle/accuracy_4_of_4‚1 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1968223601579666 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5347222089767456 jet/accuracy_4_of_6‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9867097735404968 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.16554054617881775 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0416666679084301 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.988183319568634 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *loss/val_total_loss‚41.833160325672296 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24965134263038635 particle/accuracy_1_of_5‚1 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04597701132297516 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5642856955528259 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4901565909385681 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.34340277314186096 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5659685134887695 *jet/accuracy_2_of_2‚0.1087983027100563 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9899565577507019 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 $ _timestamp‚1.788469626507405e+09 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.30000001192092896 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.524760365486145 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.35646504163742065 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8137417435646057 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5752976536750793 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4348784387111664 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5218191146850586 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07669685781002045 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 trainer/global_step‚4835 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4081568419933319 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.11777777969837189 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5380234122276306 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.28672876954078674 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5680246353149414 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5636757016181946 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5951150059700012 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5138888955116272 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 particle/accuracy_5_of_7‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5384024977684021 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9875683188438416 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1577407568693161 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3437499701976776 particle/accuracy_1_of_6‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9900316596031189 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5546320676803589 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1610453724861145 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3086419701576233 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14758478105068207 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.024611370638012886 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6875 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.43560609221458435 particle/accuracy_2_of_3‚1 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5285612344741821 6 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.270114928483963 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 particle/accuracy_6_of_7‚1 particle/accuracy_7_of_7‚0.5 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04463891685009003 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.12885595858097076 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9952331185340881 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 _runtime‚5679.694579905 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14015352725982666 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7658209204673767 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 _step‚395 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2605401575565338 jet/accuracy_5_of_7‚0 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5055035352706909 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3104563355445862 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.18008598685264587 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16252003610134125 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5652270913124084 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9992784857749939 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.44736841320991516 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5589508414268494 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.17045454680919647 *jet/accuracy_1_of_1‚0.4991779029369354 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4628470242023468 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.46081089973449707 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6754385828971863 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2202380895614624 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0416666679084301 particle/accuracy_5_of_6‚1 *"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.5 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9881494641304016 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9938775897026062 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974977374076843 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 particle/accuracy_2_of_6‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.523150622844696 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4642857015132904 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.38418903946876526 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08703155070543289 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.29629626870155334 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_7‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.12874630093574524 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07434423267841339 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_7‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5229007005691528 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5198968648910522 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02234465815126896 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.48245614767074585 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5962302088737488 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5948275923728943 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.027188941836357117 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ,jet/accuracy_3_of_6‚0.057692307978868484 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.326299786567688 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9917161464691162 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1493576020002365 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9722222685813904 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.08506228029727936 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_7_of_7‚0 particle/accuracy_2_of_7‚1 7"Purity/*FRt1SRqqt0FBt/event_purity‚0.28408083319664 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5199999809265137 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2557367980480194 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41195863485336304 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.136943519115448 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1599142849445343 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.41470590233802795 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.14952129125595093 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_4_of_7‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2613767087459564 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1827145516872406 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4468390941619873 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.49108150601387024 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983749389648438 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.8888888955116272 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9906398057937622 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03231929615139961 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.38333332538604736 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 8"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.446368932723999 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01554129458963871 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 +jet/accuracy_2_of_4‚0.39119818806648254 jet/accuracy_1_of_6‚1 particle/accuracy_4_of_6‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.40920087695121765 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.43117815256118774 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988309144973755 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8611111044883728 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4722222089767456 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15986546874046326 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.00907808355987072 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14641985297203064 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5035097599029541 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1324712634086609 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.988300621509552 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4729063808917999 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.1158900111913681 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4554263651371002 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.5685932040214539 particle/accuracy_3_of_5‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8866235017776489 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.008620689623057842 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14029325544834137 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9875598549842834 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 particle/accuracy_2_of_2‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2595468759536743 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9892221689224243 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5771604776382446 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 +jet/accuracy_2_of_3‚0.24680721759796143 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05342261865735054 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.28611114621162415 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 5 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.38112872838974 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4889514148235321 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.565019428730011 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.16091953217983246 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.2777777910232544 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21266232430934906 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.3125 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5921769142150879 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1941024363040924 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03525061905384064 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.43407928943634033 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.12423178553581238 :"Puri{U õEty/0FRt3SRqqt0FBt/SRqqt_purity‚0.41655775904655457 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7857142686843872 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5392869114875793 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00570973614230752 jet/accuracy_5_of_5‚0 (particle/f_score‚0.40541210770606995 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.006134077440947294 *jet/accuracy_1_of_5‚0.9471837878227234 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 *jet/accuracy_2_of_6‚0.5769230723381042 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9919160604476929 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.36852678656578064 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 +particle/sensitivity‚0.3135445713996887 7validation_average_jet_accuracy‚0.44616997241973877 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9858116507530212 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1363026648759842 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.018834032118320465 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5292433500289917 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1300831437110901 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.533282995223999 particle/accuracy_1_of_3‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5369405150413513 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4512481391429901 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 +particle/specificity‚0.9569346308708191 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5110954642295837 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.23268677294254303 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1219298243522644 particle/accuracy_1_of_2‚1 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.34907400608062744 *"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9021164774894714 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6391556859016418 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4586028754711151 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4166666865348816 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09978359937667847 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13723443448543549 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20338910818099976 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.17420634627342224 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4552631974220276 particle/accuracy_3_of_6‚1 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0‹4’‘‚(:† ÿÂçÔ´¶Ýˆ gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚92.689' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚23.17225 gpu.0.smClock‚1230 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XjuŽ‚(:ê ÿÂçÔÒ²´‰ network.recv‚ 1.90957e+06 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda2.out‚619.7158203125# disk.nvme1n1.out‚42822.44921875$ disk./.usageGB‚2974.797866821289 disk.rbd2.in‚0 disk.sdb2.in‚ 3.49609375 disk.md0.out‚ 597.67578125 disk.rbd0.out‚ 1.68359375 disk.sdb.out‚619.7158203125 disk.dm-0.out‚ 72696.9453125 disk.sda1.out‚0 disk.rbd2.out‚ 1.62109375 disk.sda.in‚ 3.62890625 disk.md0p1.in‚7.125$ memory_percent‚35.03830432891846* proc.memory.availableMB‚10643.32421875 disk.sda2.in‚ 3.62890625 network.sent‚ 1.1643196e+07 disk.md0.in‚7.125 disk.rbd0.in‚0.015625 disk.sda.out‚619.7158203125 disk.nvme0n1.in‚ 328970.03125 disk.md0p1.out‚ 597.67578125 proc.memory.rssMB‚ 3031.71875 disk.sdb1.in‚0 disk.sdb1.out‚0# disk.nvme0n1.out‚29874.49609375) proc.memory.percent‚18.50414276123047# disk.nvme1n1.in‚200722.94140625) disk./.usagePercent‚83.17659340595578 disk.sdb2.out‚619.7158203125 disk.sdb.in‚ 3.49609375 disk.dm-0.in‚529692.97265625 cpu‚25.188930620522278(p¸½2- _wandbruntime‚5698  _runtime‚5698ÇËmwÁ» trainer/global_step‚4849 # _timestamp‚1.78846964516928e+09 _runtime‚5698.354220149 _step‚396 $ lr-AdamW/pg1‚0.00061699812109791 $ lr-AdamW/pg2‚0.00061699812109791Tu|RÆ‘À _step‚396 $ lr-AdamW/pg1‚0.00061699812109791 $ lr-AdamW/pg2‚0.00061699812109791 trainer/global_step‚4849 # _timestamp‚1.78846964516928e+09 _runtime‚5698.354220149Œ×XK[2’- _wandbruntime‚5698  _runtime‚5698 Mâà'“ÚO LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚198131563125760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚188476854155504 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _runtime‚5698.355374324 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9645565547008 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚654 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207776895004160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚198095084317246 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55342284 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚219645186474496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 Í|å'”ßO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9618213122382 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7884696451703298e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚207776895004160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9645331878400 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 132021724 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚207777146675712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207776895004160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚198095084317246 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55342284 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚219645186474496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚198131563125760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚188476854155504 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _runtime‚5698.355374324 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9645565547008 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚654 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚90502330 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9619942630284 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90501701 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55353086 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚188514384512928 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚198170849162752 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132048416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9811661739008 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚209877200071168 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9647295904256 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9647295904256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚207818145067008 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92540778 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚219688861810176 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _step‚398 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚198170867165696 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚188514401552288 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9620171611576 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 224589194 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚207818396751872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92540780 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35149244 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 224615566 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚198134327143212 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 % _timestamp‚1.7884696462996888e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚4849 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚219689180550656 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 224615566 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92540778 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9811956539392 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92540780 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9647529586176 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 224589194 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚627 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚198170867165696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚209877224011264 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 132074786 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9647529586176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚198134573163864 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 y?ï&—éM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚198170849162752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90501701 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55353086 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚188514384512928 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚198170849162752 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132048416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9811661739008 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚209877200071168 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9647295904256 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9647295904256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚207818145067008 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92540778 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚219688861810176 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _step‚398 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚198170867165696 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚188514401552288 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9620171611576 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 224589194 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚207818396751872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92540780 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35149244 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 224615566 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚198134327143212 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 % _timestamp‚1.7884696462996888e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚4849 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚219689180550656 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 224615566 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92540778 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9811956539392 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92540780 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9647529586176 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 224589194 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚627 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚198170867165696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚209877224011264 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 132074786 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9647529586176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚198134573163864 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132074786 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35148617 _runtime‚5699.485677343 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚207818396751872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚90502330 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9619942630284 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93436253 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚200069540228954 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚200095864975360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884697021884298e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93436255 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9743547585024 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9910065709056 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 226795929 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚211919714603520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9909781381120 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9715922181806 _runtime‚5755.373833991 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93436253 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_starYè‚Юt/segment.all.allocated‚582 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚209839412560384 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚209839664231936 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚211919738543616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9716151162890 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9743547585024F{mqå'¨ßO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55882384 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93436253 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚200069540228954 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚200095864975360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884697021884298e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93436255 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9743547585024 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9910065709056 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 226795929 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚211919714603520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9909781381120 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9715922181806 _runtime‚5755.373833991 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93436253 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚209839412560384 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚209839664231936 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚211919738543616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9716151162890 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9743547585024 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35483740 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚401 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9743781253632 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚200095882978304 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 trainer/global_step‚4899 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚190353372026704 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚91366122 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93454528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚200135151012352 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚221873486831104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚91384926 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55893184 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚190390902384128 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 trainer/global_step‚4899 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 226840896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91384231 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚209880662623232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9745745292800 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚221873178576384 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚211961398573568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 133412738 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚200135169015296 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93454530 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93454530 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 133386368 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9745511610880 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _step‚402 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚211961422513664 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚200135169015296 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226867268 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 133412738 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚693 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 226840896 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚190390919423488 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚209880914308096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9911780002816 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9745745292800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35491740 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 226867268 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚209880914308096 % _timestamp‚1.7884697033579316e+09 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9912064317440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-14_#óï&«éM PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚211961398573568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 133412738 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚200135169015296 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93454530 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93454530 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 133386368 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9745511610880 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _step‚402 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚211961422513664 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚200135169015296 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226867268 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 133412738 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚693 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 226840896 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚190390919423488 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚209880914308096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9911780002816 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9745745292800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35491740 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 226867268 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚209880914308096 % _timestamp‚1.7884697033579316e+09 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9912064317440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93454528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚200135151012352 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚221873486831104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚91384926 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55893184 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚190390902384128 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 trainer/global_step‚4899 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 226840896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91384231 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚209880662623232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9745745292800 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚221873178576384’±“;n2¬- _wandbruntime‚5756  _runtime‚5756Ôý”X­Ò 1loss/SRqqt3/detection_loss‚0.9175959055962628 epoch‚4 .loss/FRt1/assignment_loss‚2.74627735828742 /loss/FRt4/assignment_loss‚3.747702980041504 /loss/FBt3/detection_loss‚0.7554102355285602 1loss/FBt1/assignment_loss‚0.06420698631554841 0loss/FBt4/assignment_loss‚0.1723147093837794 .loss/FRt4/detection_loss‚4.057084886233012 1loss/SRqqt2/detection_loss‚2.2791213945430866 1loss/FBt3/assignment_loss‚0.10923631674753885 1loss/SRqqt4/detection_loss‚2.4339610159397127 % _timestamp‚1.7884697033612902e+09 _runtime‚5756.546185288 /loss/FRt2/assignment_loss‚4.214372758865356 2loss/SRqqt4/assignment_loss‚1.1689373904218276 /loss/FRt3/detection_loss‚1.7927132419085707 _step‚403 trainer/global_step‚4899 2loss/SRqqt1/assignment_loss‚1.6408280084530513 0loss/SRqqt2/assignment_loss‚2.09560986476786 /loss/FRt1/detection_loss‚2.5089089335539403 0loss/SRqqt1/detection_loss‚1.838941044277615 .loss/FBt2/detection_loss‚3.250676655769348 /loss/FRt3/assignment_loss‚1.237817961594154 1loss/FBt2/assignment_loss‚0.09706133012077771 /loss/FRt2/detection_loss‚1.9947458136454224 /loss/FBt4/detection_loss‚2.1059355460680447 1loss/SRqqt3/assignment_loss‚1.393777280050802 /loss/FBt1/detection_loss‚2.9301716327667235 &loss/total_loss‚45.553409250879916ÅFï ®× /loss/FRt2/assignment_loss‚4.214372758865356 2loss/SRqqt4/assignment_loss‚1.1689373904218276 /loss/FRt3/detection_loss‚1.7927132419085707 «dT±½_step‚403 trainer/global_step‚4899 2loss/SRqqt1/assignment_loss‚1.6408280084530513 0loss/SRqqt2/assignment_loss‚2.09560986476786 /loss/FRt1/detection_loss‚2.5089089335539403 0loss/SRqqt1/detection_loss‚1.838941044277615 .loss/FBt2/detection_loss‚3.250676655769348 /loss/FRt3/assignment_loss‚1.237817961594154 1loss/FBt2/assignment_loss‚0.09706133012077771 /loss/FRt2/detection_loss‚1.9947458136454224 /loss/FBt4/detection_loss‚2.1059355460680447 1loss/SRqqt3/assignment_loss‚1.393777280050802 /loss/FBt1/detection_loss‚2.9301716327667235 &loss/total_loss‚45.553409250879916 1loss/SRqqt3/detection_loss‚0.9175959055962628 epoch‚4 .loss/FRt1/assignment_loss‚2.74627735828742 /loss/FRt4/assignment_loss‚3.747702980041504 /loss/FBt3/detection_loss‚0.7554102355285602 1loss/FBt1/assignment_loss‚0.06420698631554841 0loss/FBt4/assignment_loss‚0.1723147093837794 .loss/FRt4/detection_loss‚4.057084886233012 1loss/SRqqt2/detection_loss‚2.2791213945430866 1loss/FBt3/assignment_loss‚0.10923631674753885 1loss/SRqqt4/detection_loss‚2.4339610159397127 % _timestamp‚1.7884697033612902e+09 _runtime‚5756.546185288“ªBFT’¯‚(:‡ ÊÃçÔ’ž»ˆ gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚144.821' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.20525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 F±+y°‚(:î ÊÃçÔï —‰! disk.sdb2.out‚627.73583984375 disk.sdb.out‚627.73583984375 disk.sdb1.in‚0$ memory_percent‚36.08255386352539 disk.md0p1.out‚ 605.37890625 disk.rbd0.out‚ 1.68359375 disk.sda.in‚ 3.62890625# disk.nvme0n1.out‚29875.17578125 disk.sda2.in‚ 3.62890625 proc.cpu.threads‚17 disk.md0p1.in‚7.125 disk.md0.in‚7.125 disk.rbd0.in‚0.015625 disk.sda.out‚627.73583984375! disk.sda2.out‚627.73583984375# proc.memory.rssMB‚ 3030.74609375 cpu‚25.18687516293879 network.sent‚ 1.1904096e+07( disk./.usagePercent‚83.1842747774976 disk.sdb1.out‚0% disk./.usageGB‚2975.0725898742676 disk.sdb2.in‚ 3.49609375 disk.rbd2.out‚ 1.62109375 disk.sdb.in‚ 3.49609375! disk.nvme0n1.in‚ 328970.046875 disk.dm-0.in‚529693.01953125( proc.memory.availableMB‚ 10472.234375 network.recv‚ 1.935631e+06 disk.md0.out‚ 605.37890625 disk.rbd2.in‚0# disk.nvme1n1.in‚200722.97265625# disk.nvme1n1.out‚44101.48046875 disk.sda1.in‚0) proc.memory.percent‚18.49820613861084 disk.dm-0.out‚ 73976.65625 disk.sda1.out‚0(a‘±‚(:„ ÙÃçÔ±¸ ‡ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚86.502' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚21.6255 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'ú5¬{²‚(:ð ÙÃçÔÔªû‡$ disk./.usageGB‚2974.797679901123 disk.sdb2.in‚ 3.49609375 disk.sdb.in‚ 3.49609375 disk.sda.in‚ 3.62890625$ memory_percent‚36.08138561248779 disk.rbd2.in‚0 disk.sdb.out‚629.59423828125 disk.sda.out‚629.59423828125 cpu‚25.186446617033404) disk./.usagePercent‚83.17658817958971 disk.md0.out‚ 607.1640625 disk.rbd0.out‚ 1.68359375 disk.rbd2.out‚ 1.62109375 disk.sdb1.in‚0! disk.nvme0n1.in‚ 328970.046875# proc.memory.rssMB‚ 3030.74609375 network.sent‚ 1.200067e+07! disk.sdb2.out‚629.59423828125# disk.nvme1n1.out‚44324.48828125! disk.sda2.out‚629.59423828125 disk.md0p1.in‚7.125 disk.rbd0.in‚0.015625 disk.sda2.in‚ 3.62890625 disk.dm-0.in‚529693.03515625 disk.sda1.out‚0) proc.memory.percent‚18.49820613861084" disk.nvme0n1.out‚ 29875.2265625 disk.sda1.in‚0* proc.memory.availableMB‚10472.42578125 proc.cpu.threads‚17 network.recv‚ 1.942472e+06 disk.md0p1.out‚ 607.1640625 disk.md0.in‚7.125# disk.nvme1n1.in‚200722.98828125 disk.sdb1.out‚0 disk.dm-0.out‚74199.71484375„ø~œ³‚(:‘ èÃçÔ®ÙÕ‡ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚119.368' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.841999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Øz´‚(:ï èÃçÔ®¿Šˆ network.recv‚ 1.946991e+06 disk.rbd0.out‚ 1.68359375 disk.sda2.in‚ 3.62890625) proc.memory.percent‚18.49820613861084 disk.rbd2.out‚ 1.62109375! disk.nvme0n1.in‚ 328970.046875" disk.nvme0n1.out‚ 29876.0859375! disk.sda2.out‚631.18505859375 network.sent‚ 1.2006763e+07 disk.sdb1.out‚0 disk.sda.out‚631.18505859375 disk.dm-0.in‚529693.03515625 disk.dm-0.out‚74335.25390625 disk.sda1.out‚0# proc.memory.rssMB‚ 3030.74609375! disk.sdb2.out‚631.18505859375 disk.rbd2.in‚0 disk.sdb1.in‚0 disk.sda.in‚ 3.62890625 disk.sda1.in‚0 disk.md0p1.in‚7.125 disk.md0p1.out‚ 608.69140625 disk.md0.out‚ 608.69140625 disk.rbd0.in‚0.015625( proc.memory.availableMB‚ 10472.171875) disk./.usagePercent‚83.14580413679033 disk.md0.in‚7.125# disk.nvme1n1.in‚200722.98828125 disk.sdb.out‚631.18505859375 proc.cpu.threads‚17# disk.nvme1n1.out‚44459.16796875# disk./.usageGB‚2973.69669342041 disk.sdb2.in‚ 3.49609375 disk.sdb.in‚ 3.49609375$ memory_percent‚36.08293533325195 cpu‚25.18605112693246ÙúLœµ‚(:‘ ÷ÃçÔ¸Šê… gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!Ymdg¶‚(:Ü ÷ÃçÔ¯òá†" disk.nvme0n1.out‚ 29876.2734375 disk.dm-0.out‚74516 disk.sdb2.in‚ 3.49609375 disk.rbd0.out‚ 1.68359375" disk.nvme1n1.out‚ 44639.7265625! disk.nvme0n1.in‚ 328970.046875 disk.md0.in‚7.125 disk.sdb.out‚ 632.58203125 cpu‚25.185627909215917 proc.cpu.threads‚17 disk.sda.out‚ 632.58203125 disk.rbd0.in‚0.015625 disk.rbd2.in‚0) proc.memory.percent‚18.49820613861084 disk.sda.in‚ 3.62890625 disk.sda1.in‚0 disk.sdb2.out‚ 632.58203125$ memory_percent‚36.08295917510986# proc.memory.rssMB‚ 3030.74609375) disk./.usagePercent‚83.14772007992885 disk.sdb1.out‚0 disk.sda2.out‚ 632.58203125# disk.nvme1n1.in‚200722.98828125% disk./.usageGB‚2973.7652168273926 disk.md0p1.in‚7.125 disk.sdb.in‚ 3.49609375 network.recv‚ 1.950732e+06 disk.dm-0.in‚529693.03515625 disk.sda2.in‚ 3.62890625 disk.md0p1.out‚ 610.0390625 network.sent‚ 1.2009817e+07 disk.sdb1.in‚0 disk.sda1.out‚0 disk.md0.out‚ 610.0390625 disk.rbd2.out‚ 1.62109375* proc.memory.availableMB‚10472.16796875Û(Ä2·- _wandbruntime‚5808  _runtime‚5808:Ú ²Ç¸Á & lr-AdamW/pg1‚0.0006065845887383681 & lr-AdamW/pg2‚0.0006065845887383681 trainer/global_step‚4949 % _timestamp‚1.7884697548514721e+09 _runtime‚5808.036424512 _step‚404´) “Ì¹Æ & lr-AdamW/pg1‚0.0006065845887383681 & lr-AdamW/pg2‚0.0006065845887383681 trainer/global_step‚4949 % _timestamp‚1.7884697548514721e+09 _runtime‚5808.036424512 _step‚404”Yëdº2º- _wandbruntime‚5808  _runtime‚5808Ð ýïá'»ÛO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚214003937046016 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 134697628 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 $ _timestamp‚1.788469754852534e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚202060166824960 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚211901930116608 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9814089203398 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94350003 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35831952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚211901930116608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚675 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚211902181788160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚202060166824960 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 229047631 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9841763291648 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 134697628 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚224013682887168 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚202043996140662 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94350005 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 134723972 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚192229906937264 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35832627 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚94350003 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9841763291648 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚214003913105920 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚224013999543808 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10010062497792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚202060184827904 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56422484 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚192229889897904 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 229073977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚677 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 _runtime‚5808.037553927 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚211902181788160 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94350005 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚202043750120218 _step‚405 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚202060184827904 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚92255111 trainer/global_step‚4949 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9841996960256 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92254434 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 229047631 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9841763291648 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 134697628 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚224013682887168 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚202043996140662 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94350005 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 134723972 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚192229906937264 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35832627 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚94350003 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9841763291648 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚214003913105920 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚224013999543808 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10010062497792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚202060184827904 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56422484 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚192229889897904 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 229073977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚677 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 _runtime‚5808.037553927 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚211902181788160 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94350005 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚202043750120218 _step‚405 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚202060184827904 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚92255111 trainer/global_step‚4949 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9841996960256 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92254434 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 134697628 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 $ _timestamp‚1.788469754852534e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚202060166824960 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚211901930116608 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9814089203398 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94350003 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35831952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚211901930116608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚675 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚211902181788160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2•r!n²2½- _wandbruntime‚5809  _runtime‚5809Û쮾åM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9816047692592 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚214045597075968 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 134750690 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10011783931392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94368278 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9843960999424 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94368280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚211943180179456 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9843727317504 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92273224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56433284 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚211943431864320 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚202099452861952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚224057381007360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 229118970 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56433286 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚202099452861952 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 229092598 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9815818711300 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚192267420255328 trainer/global_step‚4949 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚202083238966628 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 229118970 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94368278 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 % _timestamp‚1.7884697558544118e+09 _runtime‚ 5809.04005725 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚406 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94368280 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9843727317504 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92272544 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚192267437294688 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 229092598 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9843960999424 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚214045621016064 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚202083484987280 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚202099470864896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928ÂN/ð&¿êM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9843960999424 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94368280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚211943180179456 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9843727317504 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92273224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56433284 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚211943431864320 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚202099452861952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚224057381007360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 229118970 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56433286 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚202099452861952 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 229092598 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9815818711300 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚192267420255328 trainer/global_step‚4949 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚202083238966628 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 229118970 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94368278 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 % _timestamp‚1.7884697558544118e+09 _runtime‚ 5809.04005725 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚406 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94368280 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9843727317504 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92272544 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚192267437294688 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 229092598 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9843960999424 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚214045621016064 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚202083484987280 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚202099470864896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9816047692592 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚214045597075968 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 134750690 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10011783931392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94368278 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0–Ìÿ±2À- _wandbruntime‚5809  _runtime‚5809)DªDWÁÑ /loss/FRt2/assignment_loss‚3.969783111316402 0loss/SRqqt3/assignment_loss‚1.13372548540835 /loss/FRt2/detection_loss‚2.3278672465324832 1loss/SRqqt2/detection_loss‚2.4923454207532547 epoch‚4 trainer/global_step‚4949 0loss/FRt3/assignment_loss‚1.4073606407831585 .loss/SRqqt2/assignment_loss‚1.644645832479 2loss/FBt1/assignment_loss‚0.002715217815421056 1loss/SRqqt4/detection_loss‚2.5168107495163428 1loss/FBt4/assignment_loss‚0.03157903225194657 /loss/FBt3/detection_loss‚0.5123465186550415 /loss/FRt1/detection_loss‚2.7646647836222793 /loss/FRt1/assignment_loss‚2.411791848413872 /loss/FBt2/detection_loss‚2.0894540710882707 %loss/total_loss‚43.14987354180726 _step‚407 1loss/SRqqt1/assignment_loss‚1.369760671074976 1loss/FBt3/assignment_loss‚0.05003238548854816 /loss/FBt1/detection_loss‚2.0454426084955535 $ _timestamp‚1.788469755856312e+09 _runtime‚5809.041070345 1loss/SRqqt4/assignment_loss‚2.086394294644847 /loss/FRt3/detection_loss‚1.7011499694060712 .loss/FBt4/detection_loss‚2.807409243924277 /loss/FRt4/assignment_loss‚3.876793794631958 2loss/FBt2/assignment_loss‚0.005053130424295721 /loss/FRt4/detection_loss‚3.1106311964988707 0loss/SRqqt1/detection_loss‚2.010107979701276 1loss/SRqqt3/detection_loss‚0.7820083088807616xØÇ\ÂÖ 1loss/SRqqt1/assignment_loss‚1.369760671074976 1loss/FBt3/assignment_loss‚0.05003238548854816 /loss/FBt1/detection_loss‚2.0454426084955535 $ _timestamp‚1.788469755856312e+09 _runtime‚5809.041070345 1loss/SRqqt4/assignment_loss‚2.086394294644847 /loss/FRt3/detection_loss‚1.7011499694060712 .loss/FBt4/detection_loss‚2.807409243924277 /loss/FRt4/assignment_loss‚3.876793794631958 2loss/FBt2/assignment_loss‚0.005053130424295721 /loss/FRt4/detection_loss‚3.1106311964988707 0loss/SRqqt1/detection_loss‚2.010107979701276 1loss/SRqqt3/detection_loss‚0.7820083088807616 /loss/FRt2/assignment_loss‚3.969783111316402 0loss/SRqqt3/assignment_loss‚1.13372548540835 /loss/FRt2/detection_loss‚2.3278672465324832 1loss/SRqqt2/detection_loss‚2.4923454207532547 epoch‚4 trainer/global_step‚4949 0loss/FRt3/assignment_loss‚1.4073606407831585 .loss/SRqqt2/assignment_loss‚1.644645832479 2loss/FBt1/assignment_loss‚0.002715217815421056 1loss/SRqqt4/detection_loss‚2.5168107495163428 1loss/FBt4/assignment_loss‚0.03157903225194657 /loss/FBt3/detection_loss‚0.5123465186550415 /loss/FRt1/detection_loss‚2.7646647836222793 /loss/FRt1/assignment_loss‚2.411791848413872 /loss/FBt2/detection_loss‚2.0894540710882707 %loss/total_loss‚43.14987354180726 _step‚407—4ýÌÂ(:… †ÄçÔè±Ú† gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚139.868' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.967 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹²ËëtÄ‚(:é †ÄçÔ¶»®‡ disk.rbd0.out‚ 1.68359375# proc.memory.rssMB‚ 3030.74609375 disk.sdb1.in‚0 disk.sda2.in‚ 3.62890625 disk.md0p1.in‚7.125 disk.sdb2.in‚ 3.49609375% disk./.usageGB‚2973.8900756835938" disk.nvme0n1.out‚ 29884.7421875 proc.cpu.threads‚17 disk.rbd2.out‚ 1.62109375 disk.sda.out‚ 634.513671875 disk.sda1.out‚0 disk.sda2.out‚ 634.513671875 disk.md0p1.out‚ 611.91796875% memory_percent‚36.081671714782715) proc.memory.percent‚18.49820613861084 disk.sdb.in‚ 3.49609375 disk.sdb.out‚ 634.513671875 disk.sda.in‚ 3.62890625 disk.rbd0.in‚0.015625# disk.nvme1n1.in‚200722.98828125) disk./.usagePercent‚83.15121118580572 disk.md0.out‚ 611.91796875" disk.nvme1n1.out‚ 44724.5234375* proc.memory.availableMB‚10472.37890625 disk.sdb1.out‚0 disk.md0.in‚7.125 disk.rbd2.in‚0 network.recv‚ 1.955269e+06! disk.nvme0n1.in‚ 328970.046875 disk.dm-0.in‚529693.03515625 disk.sdb2.out‚ 634.513671875 cpu‚25.18518954235708 network.sent‚ 1.2016191e+07 disk.dm-0.out‚ 74609.265625 disk.sda1.in‚0ͳȉÅ‚(:… •ÄçÔö½† gpu.0.gpu‚42.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚142.93' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.7325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çHÊÈsÆ‚(:è •ÄçÔ؃‡ disk.md0.in‚7.125 disk.sda.out‚636.04248046875 network.sent‚ 1.2123587e+07) disk./.usagePercent‚83.15147271743048! disk.sda2.out‚636.04248046875 disk.sdb2.in‚ 3.49609375 disk.rbd2.in‚0 disk.sdb.out‚636.04248046875 disk.sda1.in‚0 disk.rbd0.in‚0.015625 disk.sda.in‚ 3.62890625# proc.memory.rssMB‚ 3030.74609375) proc.memory.percent‚18.49820613861084 disk.sda1.out‚0% memory_percent‚36.080384254455566 disk.md0.out‚613.375! disk.nvme1n1.out‚ 44855.953125 disk.sdb1.in‚0 disk.md0p1.in‚7.125# disk.nvme0n1.out‚29884.82421875 disk.dm-0.out‚74740.77734375 cpu‚25.18471936000517 proc.cpu.threads‚17 network.recv‚ 1.962015e+06 disk.md0p1.out‚613.375 disk.rbd0.out‚ 1.68359375 disk.rbd2.out‚ 1.62109375 disk.sdb1.out‚0* proc.memory.availableMB‚10472.58984375 disk.sda2.in‚ 3.62890625# disk.nvme1n1.in‚200722.98828125 disk.sdb.in‚ 3.49609375! disk.nvme0n1.in‚ 328970.046875 disk.dm-0.in‚529693.03515625$ disk./.usageGB‚2973.899429321289! disk.sdb2.out‚636.04248046875˜"ŸœÇ‚(:‘ ¤ÄçÔòžî… gpu.0.gpu‚11.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}¬MNqÈ‚(:æ ¤ÄçÔ¨ü¡‡ disk.dm-0.out‚ 74875.0546875 network.sent‚ 1.2131849e+07 disk.sdb2.in‚ 3.49609375! disk.sdb2.out‚637.61865234375 disk.sdb.in‚ 3.49609375 disk.sdb1.out‚0 disk.sda2.in‚ 3.62890625) proc.memory.percent‚18.49820613861084 proc.cpu.threads‚17 disk.md0p1.out‚ 614.90625 disk.rbd2.in‚0! disk.nvme0n1.in‚ 328970.046875! disk.sda2.out‚637.61865234375 disk.sdb1.in‚0" disk.nvme0n1.out‚ 29970.9296875 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.out‚ 1.62109375) disk./.usagePercent‚83.15617378038375 disk.md0p1.in‚7.125 disk.md0.in‚7.125 disk.sda.out‚637.61865234375 disk.sdb.out‚637.61865234375$ memory_percent‚36.08040809631348) proc.memory.availableMB‚ 10472.5859375# proc.memory.rssMB‚ 3030.74609375% disk./.usageGB‚2974.0675621032715 disk.rbd0.out‚ 1.68359375# disk.nvme1n1.in‚200722.98828125 disk.nvme1n1.out‚ 44904.125 disk.sda.in‚ 3.62890625 disk.dm-0.in‚529693.03515625 network.recv‚ 1.966156e+06 disk.md0.out‚ 614.90625 disk.rbd0.in‚0.015625 cpu‚25.18433112850084¼n‡š2É- _wandbruntime‚5862  _runtime‚5862×é ’ÆÊÀ trainer/global_step‚4999 % _timestamp‚1.7884698088244767e+09 _runtime‚ 5862.00954894 _step‚408 & lr-AdamW/pg1‚0.0006013165219023359 & lr-AdamW/pg2‚0.0006013165219023359?»ËËÅ _runtime‚ 5862.00954894 _step‚408 & lr-AdamW/pg1‚0.0006013165219023359 & lr-AdamW/pg2‚0.0006013165219023359 trainer/global_step‚4999 % _timestamp‚1.7884698088244767e+09˜ ÷,E2Ì- _wandbruntime‚5862  _runtime‚5862¹Èu#ÍÜO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 136035580 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚204018452052370 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56962584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36174988 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301976576 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚226197745484288 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213964447672832 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93137572 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95263753 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10109911912448 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95263755 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚664 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95263755 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚204024486677504 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚194106407769104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95263753 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚213964447672832 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 136035580 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚194106424808464 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 136061924 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 231325679 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚662 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 278036480 trainer/global_step‚4999 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚204024468674560 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93136908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36174326 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 136061924 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚213964699344384 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9939978998272 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9911798262822 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7884698088257961e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 _step‚409 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚204024486677504 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚204024468674560 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚5862.011100847 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚226198047460864 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚216088135548416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9912027243906 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9940212666880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10109633875968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213964447672832 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93137572 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95263753 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10109911912448 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95263755 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚664 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95263755 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚204024486677504 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚194106407769104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95263753 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚213964447672832 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 136035580 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚194106424808464 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 136061924 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 231325679 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚662 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 278036480 trainer/global_step‚4999 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚204024468674560 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93136908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36174326 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 136061924 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚213964699344384 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9939978998272 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9911798262822 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7884698088257961e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552™„‹Õ2Ï- _wandbruntime‚5863  _runtime‚5863fÖ%ëì&ÐæM MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚194143938126528 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9913985733100 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 _step‚410 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36181666 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚204057940898988 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95282030 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚204057694878336 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 trainer/global_step‚4999 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 231370672 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚214005697735680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚216129795578368 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 % _timestamp‚1.7884698100967126e+09 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9913756751808 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93155050 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚226241434172928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9941943024128 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95282030 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 231344300 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136062272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚204063772714496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9941943024128 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56973384 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚214005949420544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95282028 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136088642 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231370672 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 136088642 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10111935492096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚194143955165888 _runtime‚5863.283950827 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93155713 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36Mhƒñ&ÑëM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36182327 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9942176706048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10111638594560 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56973384 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚214005949420544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95282028 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136088642 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231370672 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 136088642 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10111935492096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚194143955165888 _runtime‚5863.283950827 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93155713 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚194143938126528 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9913985733100 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 _step‚410 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36181666 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚204057940898988 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95282030 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚204057694878336 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 trainer/global_step‚4999 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 231370672 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚214005697735680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚216129795578368 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 % _timestamp‚1.7884698100967126e+09 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9913756751808 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93155050 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚226241434172928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9941943024128 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95282030 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 231344300 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136062272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚204063772714496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9941943024128 Fg92Ý- _wandbruntime‚5917  _runtime‚59179fýµÇÞÁ _step‚412 & lr-AdamW/pg1‚0.0005960090489581174 & lr-AdamW/pg2‚0.0005960090489581174 trainer/global_step‚5049 % _timestamp‚1.7884698639154632e+09 _runtime‚5917.100417937úo×ãÌßÆ trainer/global_step‚5049 % _timestamp‚1.7884698639154632e+09 _runtime‚5917.100417937 _step‚412 & lr-AdamW/pg1‚0.0005960090489581174 & lr-AdamW/pg2‚0.0005960090489581174œò¨š52à- _wandbruntime‚5917  _runtime‚5917u?ÌUè'áâO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137399876 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96177505 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚216027216900608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚205988788527104 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚205992907964078 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10038428373504 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚205992661943634 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚96177503 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10209885528064 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10209586520064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚205988788527104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10009736303330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 137373532 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚216027216900608 % _timestamp‚1.7884698639164677e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚195982925640304 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚216026965229056 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚5917.101474382 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10038428373504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57502684 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10009965284414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96177505 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 137399876 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚94024044 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10038194704896 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36521362 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚218172310110720 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚205988770524160 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚228382219578880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚228381896630784 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 233551035 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57502682 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10038194704896 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36521362 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚218172310110720 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚205988770524160 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚228382219578880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚228381896630784 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 233551035 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57502682 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚205988788527104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10009736303330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 137373532 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚216027216900608 % _timestamp‚1.7884698639164677e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚195982925640304 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚216026965229056 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2336686086²#“2ã- _wandbruntime‚5918  _runtime‚5918˼QóéäêM VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10040158730752 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96195780 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚206028074564096 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 233622374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96195778 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10011694792316 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚216068466976768 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚196020473037088 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 137426594 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94041568 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36528766 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚228425889422848 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10011923773608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137400224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 5918.22136333 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚684 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10211871401984 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57513486 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚218213994080768 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 233596002 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_sîæùtreams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚196020455997728 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚206028074564096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233622374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10040392412672 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚218214018020864 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 trainer/global_step‚5049 _step‚414 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582ÿ@¿õ&åïM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 137400224 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94042252 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚216068215291904 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚206028056561152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 137426594 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 233596002 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚96195780 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 trainer/global_step‚5049 _step‚414 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10040158730752 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96195780 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚206028074564096 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 233622374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96195778 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10011694792316 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚216068466976768 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚196020473037088 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 137426594 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94041568 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36528766 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚228425889422848 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10011923773608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137400224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 5918.22136333 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚684 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10211871401984 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57513486 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚218213994080768 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 233596002 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚196020455997728 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚206028074564096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233622374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10040392412672 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚218214018020864 B¶ÌóÆ & lr-AdamW/pg1‚0.0005906632759751197 & lr-AdamW/pg2‚0.0005906632759751197 trainer/global_step‚5099 % _timestamp‚1.7884699199777954e+09 _runtime‚5973.162745449 _step‚416 O12ô- _wandbruntime‚5973  _runtime‚5973ì+=võâO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚220256508613120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10136644080128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚94911096 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10136410411520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚218089482785280 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97091253 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚230565952428032 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚220256532553216 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 138711484 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚688 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10107674343838 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36869000 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚230566264890368 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10309732337152 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 138711484 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚207953090376704 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚207953090376704 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚197859443511504 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚207967363875786 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 235829083 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚197859460550864 QBDeviceStatsMonitp Mÿr or.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚207953072373760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 trainer/global_step‚5099 _runtime‚5973.163955234 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 138737828 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58042782 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97091255 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10136644080128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97091255 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 235802737 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚686 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36868314 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94911784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58042784 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582}V í'öçO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10136644080128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚94911096 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10136410411520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚218089482785280 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97091253 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚230565952428032 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚220256532553216 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 138711484 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚688 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10107674343838 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36869000 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚230566264890368 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10309732337152 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 138711484 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚207953090376704 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚207953090376704 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚197859443511504 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚207967363875786 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 235829083 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚197859460550864 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚207953072373760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 trainer/global_step‚5099 _runtime‚5973.163955234 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 138737828 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58042782 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97091255 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10136644080128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97091255 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 235802737 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚686 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36868314 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94911784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58042784 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚220256508613120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0¡D ˆ!2÷- _wandbruntime‚5974  _runtime‚5974®KWò&øìM YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10138374437376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 _step‚418 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚197896973868928 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 138738176 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 235847704 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97109530 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚208006852722404 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138764546 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58053584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97109528 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10311726675456 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10109632832824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚218130984532992 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36876121 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884699209699252e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 138738176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚207992358410752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚207992376413696 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94929032 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚230609943198720 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚197896990908288 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _runtime‚5974.155884661 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235874076 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚220298216523264 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚220298192583168 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10109861814116 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚208006606701752 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚207992376413696 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚218130732848128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58053586 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10138608119296 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10138374437376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 138764546 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚5099 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10138608119296 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0Ú쀔Z%ùñM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97109530 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚218130984532992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚5099 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10138608119296 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10138374437376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 _step‚418 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚197896973868928 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 138738176 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 235847704 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97109530 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚208006852722404 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138764546 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58053584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97109528 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10311726675456 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10109632832824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚218130984532992 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36876121 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884699209699252e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 138738176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚207992358410752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚207992376413696 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94929032 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚230609943198720 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚197896990908288 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _runtime‚5974.155884661 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235874076 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚220298216523264 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚220298192583168 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10109861814116 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚208006606701752 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚207992376413696 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚218130732848128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58053586 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10138608119296 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10138374437376 @:DeviceStatsMonitor.on_train_batch_end/num_sd©Ðync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 138764546 B"u‚‚(:ê ÉÅçÔ“¢¡ˆ disk.md0.out‚ 632.0234375 disk.rbd0.in‚0.015625 disk.sdb1.in‚0 disk.dm-0.out‚ 77191.4140625 disk.md0p1.in‚7.125$ memory_percent‚36.07978820800781% disk./.usageGB‚2974.9560470581055! disk.sdb2.out‚655.41943359375 disk.rbd2.in‚0! disk.nvme1n1.out‚ 45675.921875 disk.sdb.in‚ 3.49609375 disk.dm-0.in‚ 529698.953125 disk.sda2.in‚ 3.62890625 proc.cpu.threads‚17 network.sent‚ 1.2499827e+07 disk.md0.in‚7.125 disk.sdb.out‚655.41943359375& proc.memory.availableMB‚ 10472.6875# proc.memory.rssMB‚ 3030.74609375) proc.memory.percent‚18.49820613861084 cpu‚25.180037820535887! disk.sda2.out‚655.41943359375 disk.sdb1.out‚0# disk.nvme0n1.in‚328975.66796875 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚ 2.01876e+06 disk.rbd2.out‚ 1.62109375# disk.nvme1n1.in‚200723.28515625 disk.rbd0.out‚ 1.68359375 disk.sda.out‚655.41943359375" disk.nvme0n1.out‚ 31515.4921875) disk./.usagePercent‚83.18101619158075 disk.sdb2.in‚ 3.49609375 disk.sda.in‚ 3.62890625 disk.md0p1.out‚ 632.0234375²w*·2ƒ- _wandbruntime‚6029  _runtime‚6029ú怺Ç„Á trainer/global_step‚5149 % _timestamp‚1.7884699759735765e+09 _runtime‚6029.158617701 _step‚420 & lr-AdamW/pg1‚0.0005852803170044195 & lr-AdamW/pg2‚0.0005852803170044195 Ç£âÌ…Æ & lr-AdamW/pg1‚0.0005852803170044195 & lr-AdamW/pg2‚0.0005852803170044195 trainer/global_step‚5149 % _timestamp‚1.7884699759735765e+09 _runtime‚6029.158617701 _step‚420¤.îh2†- _wandbruntime‚6029  _runtime‚6029¾v”Tç'‡áO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95797587 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚222340731055616 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚209917374223360 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98005003 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚209917392226304 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58582882 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58582884 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚209941573767050 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚421 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98005003 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚232750009356288 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10234859786752 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10234859786752 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚685 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚95798272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 140049436 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10409302240768 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚220152000341504 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282230784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10205612384346 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚6029.159975107 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10409584471552 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚220152000341504 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282230784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10205612384346 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚6029.159975107 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10409584471552 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98005003 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚209917392226304 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58582882 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58582884 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚209941573767050 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚421 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98005003 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚232750009356288 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10234859786752 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10234859786752 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚685 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚95798272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104¥’šU5’‰‚(:‡ ØÅçÔòò¼‡ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚130.841' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.71025 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ª=tŠ‚(:é ØÅçÔž’ùˆ cpu‚25.17962770262641# disk.nvme1n1.out‚45676.26171875% memory_percent‚36.080265045166016) disk./.usagePercent‚83.18450271105475 disk.dm-0.in‚ 529698.953125 disk.md0.out‚ 634.015625# disk.nvme1n1.in‚200723.28515625( proc.memory.availableMB‚ 10472.609375 disk.rbd2.in‚0# disk.nvme0n1.in‚328975.66796875 disk.sdb2.out‚657.4814453125 proc.cpu.threads‚17 disk.sda1.in‚0 disk.md0p1.in‚7.125 disk.sda.in‚ 3.62890625 disk.dm-0.out‚ 77477.4140625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda.out‚657.4814453125# proc.memory.rssMB‚ 3030.74609375 disk.sda2.in‚ 3.62890625 disk.md0p1.out‚ 634.015625 disk.rbd2.out‚ 1.62109375) proc.memory.percent‚18.49820613861084 disk.sdb.out‚657.4814453125 network.recv‚ 2.023178e+06# disk.nvme0n1.out‚31801.15234375 disk.sda2.out‚657.4814453125 disk.sdb2.in‚ 3.49609375 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 1.68359375 network.sent‚ 1.2516511e+07$ disk./.usageGB‚2975.080741882324 disk.sda1.out‚0 disk.md0.in‚7.125 disk.sdb.in‚ 3.49609375#ýl2‹- _wandbruntime‚6030  _runtime‚6030ÉÁÇŒéM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58593686 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58593684 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚209956660260352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚676 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98023280 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37222026 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚209981062613460 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚678 ^KDeviceStatsMonitor.on_t)xæ;(rain_batch_end/inactive_split_bytes.small_pool.freed‚10411294341632 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 238099406 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚199773508779488 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37221350 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚232794002070528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 140102498 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 140102498 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚220193502089216 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 140076128 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 140076128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 trainer/global_step‚5149 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98023278 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 # _timestamp‚1.78846997704923e+09 _step‚422 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚95815712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚220193502089216 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚222382415025664 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 238125778 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95815034 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚209956678263296 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚209956678263296 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10207799854624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238125778 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚199773491740128 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚232793685427200 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚209981308634112 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10236823825920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚222382391085568 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10411587044864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚220193250404352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944«Z¯Pô&îM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 # _timestamp‚1.78846997704923e+09 _step‚422 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚95815712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚220193502089216 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚222382415025664 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 238125778 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95815034 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚209956678263296 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚209956678263296 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10207799854624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238125778 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚199773491740128 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚232793685427200 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚209981308634112 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10236823825920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚222382391085568 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10411587044864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚220193250404352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58593686 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58593684 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚209956660260352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚676 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98023280 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37222026 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚209981062613460 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚678 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10411294341632 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 238099406 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚199773508779488 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37221350 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚232794002070528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 140102498 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 140102498 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚220193502089216 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 140076128 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 140076128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 trainer/global_step‚5149 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98023278 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0¦¿dÇÎ2Ž- _wandbruntime‚6030  _runtime‚6030­Î"XÒ 2loss/SRqqt2/assignment_loss‚1.6809462231618386 /loss/FRt2/detection_loss‚1.8304678611177951 /loss/FRt3/detection_loss‚1.5292860339617445 0loss/FRt1/assignment_loss‚3.2120751092831292 /loss/FRt2/assignment_loss‚3.275176547765732 0loss/SRqqt2/detection_loss‚2.340691879943565 % _timestamp‚1.7884699770535643e+09 /loss/FBt4/assignment_loss‚0.488888440156538 1loss/SRqqt4/detection_loss‚3.1275821954011915 .loss/FBt2/detection_loss‚2.179545106490453 /loss/FBt3/detection_loss‚0.5350316081002549 /loss/FRt4/assignment_loss‚4.426096577393381 2loss/SRqqt4/assignment_loss‚1.7676303878426551 .loss/FRt1/detection_loss‚2.874454584386614 1loss/SRqqt1/detection_loss‚2.0713611720381557 /loss/FBt1/detection_loss‚2.6147764126459756 trainer/global_step‚5149 _runtime‚6030.238418588 0loss/FBt1/assignment_loss‚0.2707673679542495 0loss/FBt3/assignment_loss‚0.0442630723930949 .loss/FBt4/detection_loss‚2.328917457507207 2loss/SRqqt3/assignment_loss‚0.8759921041535254 1loss/FBt2/assignment_loss‚0.18162243712875656 .loss/FRt4/detection_loss‚3.552256941795349 1loss/SRqqt3/detection_loss‚0.8066623782778674 _step‚423 0loss/FRt3/assignment_loss‚1.2662865122159321 epoch‚4 2loss/SRqqt1/assignment_loss‚0.9875998983310686 %loss/total_loss‚44.26837830944608°­h]× 0loss/FRt1/assignment_loss‚3.2120751092831292 /loss/FRt2/assignment_loss‚3.275176547765732 0loss/SRqqt2/detection_loss‚2.340691879943565 % _timestamp‚1.7884699770535643e+09 /loss/FBt4/assignment_loss‚0.488888440156538 1loss/SRqqt4/detection_loss‚3.1275821954011915 .loss/FBt2/detection_loss‚2.179545106490453 /loss/FBt3/detection_loss‚0.5350316081002549 /loss/FRt4/assignment_loss‚4.426096577393381 2loss/SRqqt4/assignment_loss‚1.7676303878426551 .loss/FRt1/detection_loss‚2.874454584386614 1loss/SRqqt1/detection_loss‚2.0713611720381557 /loss/FBt1/detection_loss‚2.6147764126459756 trainer/global_step‚5149 _runtime‚6030.238418588 0loss/FBt1/assignment_loss‚0.2707673679542495 0loss/FBt3/assignment_loss‚0.0442630723930949 .loss/FBt4/detection_loss‚2.328917457507207 2loss/SRqqt3/assignment_loss‚0.8759921041535254 1loss/FBt2/assignment_loss‚0.18162243712875656 .loss/FRt4/detection_loss‚3.552256941795349 1loss/SRqqt3/detection_loss‚0.8066623782778674 _step‚423 0loss/FRt3/assignment_loss‚1.2662865122159321 epoch‚4 2loss/SRqqt1/assignment_loss‚0.9875998983310686 %loss/total_loss‚44.26837830944608 2loss/SRqqt2/assignment_loss‚1.6809462231618386 /loss/FRt2/detection_loss‚1.8304678611177951 /loss/FRt3/detection_loss‚1.5292860339617445§wîvL‘‚(:… çÅçÔæ»Ý‡ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚128.056' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Nÿöo’‚(:ä çÅçÔæô¢‰% disk./.usageGB‚2973.9140815734863 disk.md0p1.in‚7.125 disk.rbd2.in‚0 disk.sdb.in‚ 3.49609375 disk.sdb.out‚ 659.32421875 cpu‚25.179304962677136 network.recv‚ 2.029018e+06 disk.sdb1.out‚0 proc.cpu.threads‚17# disk.nvme1n1.out‚45676.37890625 disk.sda.in‚ 3.62890625 disk.sda.out‚ 659.32421875# disk.nvme0n1.in‚328975.66796875 disk.dm-0.in‚ 529698.953125) proc.memory.percent‚18.49820613861084 network.sent‚ 1.2611757e+07 disk.md0.in‚7.125# disk.nvme1n1.in‚200723.28515625 disk.sdb1.in‚0 disk.md0.out‚ 635.7890625 disk.md0p1.out‚ 635.7890625 disk.sdb2.in‚ 3.49609375 disk.rbd0.out‚ 1.68359375% memory_percent‚36.080360412597656' proc.memory.availableMB‚ 10472.59375# proc.memory.rssMB‚ 3030.74609375 disk.sda1.in‚0 disk.sda2.in‚ 3.62890625 disk.rbd0.in‚0.015625 disk.dm-0.out‚ 77706.5859375) disk./.usagePercent‚83.15188240053436 disk.sda1.out‚0 disk.sda2.out‚ 659.32421875 disk.sdb2.out‚ 659.32421875 disk.rbd2.out‚ 1.62109375# disk.nvme0n1.out‚32030.20703125sä¢ ‘“‚(:† öÅçÔÕëî‡ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚135.802' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.9505 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n²ê {”‚(:ð öÅçÔå銈 disk.sda1.out‚0! disk.sdb2.out‚660.84423828125 disk.md0.in‚7.125 disk.sda.in‚ 3.62890625# disk.nvme0n1.in‚328975.66796875* proc.memory.availableMB‚10472.82421875 proc.cpu.threads‚17 network.sent‚ 1.2696141e+07 network.recv‚ 2.035857e+06# disk.nvme1n1.out‚45676.91015625 disk.sdb.in‚ 3.49609375 disk.sda.out‚660.84423828125 disk.dm-0.in‚ 529698.953125 disk.md0p1.out‚ 637.2421875 disk.sdb2.in‚ 3.49609375 disk.rbd2.out‚ 1.62109375 cpu‚25.178832947860737$ disk./.usageGB‚2973.749465942383 disk.rbd0.in‚0.015625 disk.sdb.out‚660.84423828125 disk.sda1.in‚0$ memory_percent‚36.07895374298096# proc.memory.rssMB‚ 3030.74609375) disk./.usagePercent‚83.14727967859172 disk.rbd0.out‚ 1.68359375 disk.sda2.in‚ 3.62890625! disk.sda2.out‚660.84423828125 disk.md0p1.in‚7.125) proc.memory.percent‚18.49820613861084 disk.sdb1.in‚0# disk.nvme0n1.out‚32225.82421875 disk.dm-0.out‚ 77902.734375 disk.md0.out‚ 637.2421875 disk.rbd2.in‚0# disk.nvme1n1.in‚200723.28515625 disk.sdb1.out‚0Iø“‘•‚(:† …ÆçÔ»¼ö… gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚140.478' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.1195 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l@fez–‚(:ï …ÆçÔçó˜‡! disk.sdb2.out‚662.60205078125 disk.md0.in‚7.125 disk.dm-0.in‚ 529698.953125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚ 3.62890625# proc.memory.rssMB‚ 3030.74609375$ disk./.usageGB‚2973.705821990967 disk.rbd0.out‚ 1.68359375 disk.sdb.in‚ 3.49609375$ memory_percent‚36.08353137969971 disk.dm-0.out‚ 77946.171875 disk.md0p1.out‚ 638.92578125 disk.sdb2.in‚ 3.49609375# disk.nvme1n1.in‚200723.28515625# disk.nvme1n1.out‚45676.98828125 disk.sdb1.out‚0 disk.md0p1.in‚7.125) proc.memory.percent‚18.49820613861084 cpu‚25.1783604237283 disk.sda.out‚662.60205078125# disk.nvme0n1.in‚328975.66796875* proc.memory.availableMB‚10472.07421875 proc.cpu.threads‚17 disk.sdb.out‚662.60205078125 disk.sdb1.in‚0# disk.nvme0n1.out‚32269.18359375 network.sent‚ 1.2702087e+07 network.recv‚ 2.040134e+06 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.rbd2.out‚ 1.62109375 disk.sda.in‚ 3.62890625! disk.sda2.out‚662.60205078125( disk./.usagePercent‚83.1460593754437 disk.md0.out‚ 638.92578125‘¨ßÅ2—- _wandbruntime‚6080  _runtime‚6080®p½Ç˜Á & lr-AdamW/pg1‚0.0005798612938465987 & lr-AdamW/pg2‚0.0005798612938465987 trainer/global_step‚5199 % _timestamp‚1.7884700273723736e+09 _runtime‚6080.557341024 _step‚424‚‹´Ì™Æ _step‚424 & lr-AdamW/pg1‚0.0005798612938465987 & lr-AdamW/pg2‚0.0005798612938465987 trainer/global_step‚5199 % _timestamp‚1.7884700273723736e+09 _runtime‚6080.557341024¨\“»2š- _wandbruntime‚6080  _runtime‚6080†ÖfO€›âO F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98918753 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚6080.558120378 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚211881694075904 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚211881676072960 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98918755 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37562143 % _timestamp‚1.7884700273731196e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98918753 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98918755 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚211881694075904 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚211916029678758 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10509106643968 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10332841824768 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10333075493376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10332841824768 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚224424929558016 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 trainer/global_step‚5199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 141413732 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚679 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10509388874752 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚425 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59122984 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚201612496293264 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚201612479253904 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 141413732 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚677 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDevi}³–Úh ceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚211881676072960 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚96685127 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141387388 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10333075493376 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚222214517897728 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10303550424854 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚222214517897728 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10303779405938 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚211916029678758 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10509106643968 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10332841824768 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10333075493376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10332841824768 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚224424929558016 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 trainer/global_step‚5199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 141413732 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚679 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10509388874752 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚425 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59122984 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚201612496293264 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚201612479253904 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 141413732 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚677 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚211881676072960 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚96685127 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141387388 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10333075493376 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚222214517897728 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10303550424854 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚222214517897728 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10303779405938 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98918753 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚6080.558120378 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚211881694075904 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚211881676072960 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98918755 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37562143 % _timestamp‚1.7884700273731196e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98918753 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457©8–™³2- _wandbruntime‚6081  _runtime‚6081`Spèò&žìM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚211920980112896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96702066 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 240377480 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10334805850624 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 141414080 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚211920962109952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59133784 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141440450 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚224466613528064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚201650009611328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚224466589587968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98937030 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚96702747 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98937030 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98937028 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98937028 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚211955764545820 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚211920980112896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10511119073792 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 trainer/global_step‚5199 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 141414080 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37568282 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10511405485568 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10334805850624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚211920962109952 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10335039532544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37568961 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚211955518525168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 % _timestamp‚1.7884700283371255e+09 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 240377480 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _runtime‚6081.522708321 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 240351108 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10335039532544 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚681 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚426 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚201650026650688 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 240351108 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚234977708661760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10305737895132 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚222256019645440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚679 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96702066 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 240377480 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10334805850624 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 141414080 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚211920962109952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59133784 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141440450 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚224466613528064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚201650009611328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚224466589587968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98937030 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚96702747 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98937030 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98937028 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98937028 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚211955764545820 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚211920980112896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10511119073792 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 trainer/global_step‚5199 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 141414080 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37568282 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10511405485568 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10334805850624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚211920962109952 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10335039532544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37568961 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚211955518525168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 % _timestamp‚1.7884700283371255e+09 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 240377480 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _runtime‚6081.522708321 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 240351108 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10335039532544 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚681 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚426 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚201650026650688 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 240351108 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚234977708661760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10305737895132 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment]ñäx“.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚222256019645440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚679 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ª|d¿2 - _wandbruntime‚6081  _runtime‚6081,â¤`¡Ú .loss/FRt1/detection_loss‚3.070440952266966 1loss/SRqqt4/detection_loss‚2.8513637284437814 epoch‚4 /loss/FRt4/detection_loss‚3.0459215905931263 % _timestamp‚1.7884700283390799e+09 0loss/FRt1/assignment_loss‚3.2102970821516856 2loss/SRqqt1/assignment_loss‚1.4307004251377657 1loss/FBt3/assignment_loss‚0.04085763826001596 1loss/FBt4/assignment_loss‚0.22495321858948752 /loss/FBt2/detection_loss‚2.0156890664781844 _runtime‚6081.523772626 2loss/SRqqt2/assignment_loss‚1.7429401906828086 1loss/FBt2/assignment_loss‚0.09198124711632463 0loss/FRt4/assignment_loss‚3.3849224933871516 .loss/FBt4/detection_loss‚2.379546526405546 0loss/FRt2/assignment_loss‚3.1856412913488303 0loss/FRt3/assignment_loss‚1.4803462866423787 /loss/FRt2/detection_loss‚1.8518340117399297 .loss/FBt1/detection_loss‚2.452352932521275 trainer/global_step‚5199 2loss/SRqqt4/assignment_loss‚1.6974335887158911 1loss/FBt1/assignment_loss‚0.20479082498827342 /loss/FRt3/detection_loss‚1.5469619655522748 1loss/SRqqt1/detection_loss‚1.8332579481601714 1loss/SRqqt2/detection_loss‚2.3097855786482495 1loss/SRqqt3/detection_loss‚0.9279722234655526 /loss/FBt3/detection_loss‚0.5639433506006996 _step‚427 %loss/total_loss‚42.73602248208246 2loss/SRqqt3/assignment_loss‚1.1920883201860957`b¾Êe¢ß 0loss/FRt2/assignment_loss‚3.1856412913488303 0loss/FRt3/assignment_loss‚1.4803462866423787 /loss/FRt2/detection_loss‚1.8518340117399297 .loss/FBt1/detection_loss‚2.452352932521275 trainer/global_step‚5199 2loss/SRqqt4/assignment_loss‚1.6974335887158911 1loss/FBt1/assignment_loss‚0.20479082498827342 /loss/FRt3/detection_loss‚1.5469619655522748 1loss/SRqqt1/detection_loss‚1.8332579481601714 1loss/SRqqt2/detection_loss‚2.3097855786482495 1loss/SRqqt3/detection_loss‚0.9279722234655526 /loss/FBt3/detection_loss‚0.5639433506006996 _step‚427 %loss/total_loss‚42.73602248208246 2loss/SRqqt3/assignment_loss‚1.1920883201860957 .loss/FRt1/detection_loss‚3.070440952266966 1loss/SRqqt4/detection_loss‚2.8513637284437814 epoch‚4 /loss/FRt4/detection_loss‚3.0459215905931263 % _timestamp‚1.7884700283390799e+09 0loss/FRt1/assignment_loss‚3.2102970821516856 2loss/SRqqt1/assignment_loss‚1.4307004251377657 1loss/FBt3/assignment_loss‚0.04085763826001596 1loss/FBt4/assignment_loss‚0.22495321858948752 /loss/FBt2/detection_loss‚2.0156890664781844 _runtime‚6081.523772626 2loss/SRqqt2/assignment_loss‚1.7429401906828086 1loss/FBt2/assignment_loss‚0.09198124711632463 0loss/FRt4/assignment_loss‚3.3849224933871516 .loss/FBt4/detection_loss‚2.379546526405546«óÄkDw£‚(:ì ”ÆçÔžªÔ‡ proc.cpu.threads‚17 disk.dm-0.in‚ 529698.953125 disk.sdb2.out‚664.0947265625 network.sent‚ 1.2708101e+07 disk.sda.in‚ 3.62890625 disk.sda2.out‚664.0947265625 disk.md0p1.out‚ 640.3515625 disk.sdb2.in‚ 3.49609375 disk.rbd0.out‚ 1.68359375 disk.sdb.in‚ 3.49609375) disk./.usagePercent‚83.14449274554791 disk.dm-0.out‚78016.66015625# proc.memory.rssMB‚ 3030.74609375) proc.memory.percent‚18.49820613861084# disk.nvme1n1.out‚45677.31640625 disk.sdb.out‚664.0947265625 disk.sdb1.in‚0 disk.sda1.in‚0% disk./.usageGB‚2973.6497917175293 disk.rbd0.in‚0.015625# disk.nvme1n1.in‚200723.28515625 disk.nvme0n1.out‚ 32339.34375 disk.sda1.out‚0 disk.sda2.in‚ 3.62890625 cpu‚25.177919536962733 network.recv‚ 2.044254e+06 disk.sdb1.out‚0# disk.nvme0n1.in‚328975.66796875 disk.md0p1.in‚7.125 disk.md0.out‚ 640.3515625$ memory_percent‚36.08376979827881 disk.rbd2.in‚0 disk.rbd2.out‚ 1.62109375 disk.sda.out‚664.0947265625 disk.md0.in‚7.125* proc.memory.availableMB‚10472.03515625³Ý|‘¤‚(:† ”ÆçÔDZȇ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚116.306' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.0765 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëP%ß’¥‚(:‡ £ÆçÔήã… gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚138.919' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.72975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{ Nj¦‚(:ß £ÆçÔ¿äÞ† disk.sdb.in‚ 3.49609375 disk.dm-0.out‚ 78070.15625! disk.sda2.out‚666.01123046875 disk.rbd0.out‚ 1.68359375$ memory_percent‚36.08295917510986) proc.memory.percent‚18.49820613861084 proc.cpu.threads‚17 disk.sda.out‚666.01123046875 disk.dm-0.in‚ 529698.953125 disk.sda1.out‚0 disk.sda2.in‚ 3.62890625 disk.nvme1n1.out‚ 45677.65625 disk.sda.in‚ 3.62890625# proc.memory.rssMB‚ 3030.74609375# disk.nvme1n1.in‚200723.28515625 disk.sdb.out‚666.01123046875 disk.rbd0.in‚0.015625# disk.nvme0n1.in‚328975.66796875$ disk./.usageGB‚2973.814968109131! disk.sdb2.out‚666.01123046875 disk.md0.out‚642.1875 disk.sda1.in‚0 disk.sdb2.in‚ 3.49609375 disk.rbd2.out‚ 1.62109375* proc.memory.availableMB‚10472.16796875 network.sent‚ 1.2814728e+07) disk./.usagePercent‚83.14911114658877 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0p1.in‚7.125 network.recv‚ 2.050403e+06 disk.nvme0n1.out‚32392.5 disk.md0p1.out‚642.1875 disk.md0.in‚7.125 disk.rbd2.in‚0 cpu‚25.177574068839835Û™Ú §‚(:… ²ÆçÔÈËÿ… gpu.0.gpu‚10.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚86.502' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚21.6255 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒL¥}¨‚(:ò ²ÆçÔ÷Ý€ˆ network.recv‚ 2.054854e+06 disk.sdb.in‚ 3.49609375 disk.sdb1.in‚0) disk./.usagePercent‚83.15621921777043# disk.nvme1n1.in‚200723.28515625! disk.sda2.out‚667.28564453125 disk.md0p1.in‚7.125$ memory_percent‚36.08534336090088* proc.memory.availableMB‚10471.77734375 cpu‚25.177171106205464 disk.md0.in‚7.125 network.sent‚ 1.2823315e+07 disk.sdb.out‚667.28564453125 disk.sda.in‚ 3.62890625! disk.sdb2.out‚667.28564453125 proc.cpu.threads‚17# disk.nvme1n1.out‚45678.18359375 disk.dm-0.in‚529698.98828125 disk.sda1.out‚0 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 1.62109375# proc.memory.rssMB‚ 3030.74609375) proc.memory.percent‚18.49820613861084 disk.rbd0.out‚ 1.68359375 disk.rbd2.in‚0 disk.sdb1.out‚0! disk.nvme0n1.in‚ 328975.703125 disk.dm-0.out‚ 78157.09375 disk.sda2.in‚ 3.62890625 disk.sdb2.in‚ 3.49609375% disk./.usageGB‚2974.0691871643066 disk.sda.out‚667.28564453125# disk.nvme0n1.out‚32551.66015625 disk.sda1.in‚0 disk.md0p1.out‚ 643.41015625 disk.md0.out‚ 643.41015625H:Éa2©- _wandbruntime‚6134  _runtime‚6134Õ8aǪÁ & lr-AdamW/pg2‚0.0005744073358179609 trainer/global_step‚5249 % _timestamp‚1.7884700815924215e+09 _runtime‚6134.777371464 _step‚428 & lr-AdamW/pg1‚0.0005744073358179609{ÐìaÌ«Æ & lr-AdamW/pg2‚0.0005744073358179609 trainer/global_step‚5249 % _timestamp‚1.7884700815924215e+09 _runtime‚6134.777371464 _step‚428 & lr-AdamW/pg1‚0.0005744073358179609¬Ô£b¾2¬- _wandbruntime‚6134  _runtime‚6134÷YIÖè'­âO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 242557843 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚224277035453952 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10431057531392 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚706 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚203489014164464 _runtime‚6134.778480729 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10401488465362 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10431291200000 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10431291200000 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚213890485590466 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59663084 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚97570107 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 trainer/global_step‚5249 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99832505 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚97570813 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 142751684 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10431057531392 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99832503 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚237118044083200 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚213845995925504 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚224277287125504 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚237118360739840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 142751684 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚213845977922560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚226509128060416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚224277035453952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59663082 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10609232679424 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 142725340 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚203488997125104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99832503 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 142725340 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 242584189 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚213845995925504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚213890731610910 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10401717446446 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚99832505 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 % _timestamp‚1.7884700815934632e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚226509104120320 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10608939962880 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309Ùšºí'®çO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 242557843 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚224277035453952 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10431057531392 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚706 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚203489014164464 _runtime‚6134.778480729 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10401488465362 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10431291200000 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10431291200000 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚213890485590466 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59663084 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚97570107 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 trainer/global_step‚5249 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99832505 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚97570813 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 142751684 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10431057531392 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99832503 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚237118044083200 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚213845995925504 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚224277287125504 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚237118360739840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 142751684 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚213845977922560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚226509128060416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚224277035453952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59663082 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10609232679424 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 142725340 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚203488997125104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99832503 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 142725340 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 242584189 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚213845995925504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚213890731610910 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10401717446446 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚99832505 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 % _timestamp‚1.7884700815934632e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚226509104120320 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10608939962880 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309­÷Qׯ‚(:… ÁÆçÔÊŠÓ… gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚144.212' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.053 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èlM–y°‚(:î ÁÆçÔªíׇ network.recv‚ 2.058121e+06 disk.sda2.out‚669.3583984375 proc.cpu.threads‚17 disk.sda.in‚ 3.62890625 disk.dm-0.in‚529698.98828125 disk.rbd0.out‚ 1.68359375 disk.rbd2.in‚0) disk./.usagePercent‚83.15635051688547% disk./.usageGB‚2974.0738830566406 disk.sdb.out‚669.3583984375 disk.sdb1.in‚0 disk.sda.out‚669.3583984375 disk.md0p1.in‚7.125 disk.rbd0.in‚0.015625 network.sent‚ 1.2826203e+07 disk.dm-0.out‚78421.12890625 disk.md0.in‚7.125# proc.memory.rssMB‚ 3030.74609375! disk.nvme0n1.in‚ 328975.703125 disk.sdb2.in‚ 3.49609375 disk.md0.out‚ 645.41015625) proc.memory.percent‚18.49820613861084 disk.sdb1.out‚0! disk.nvme0n1.out‚ 32742.828125# disk.nvme1n1.out‚45678.30078125 cpu‚25.17678341947567 disk.rbd2.out‚ 1.62109375 disk.sdb.in‚ 3.49609375 disk.sda1.out‚0 disk.sda2.in‚ 3.62890625# disk.nvme1n1.in‚200723.28515625$ memory_percent‚36.08081340789795* proc.memory.availableMB‚10472.51953125 disk.sda1.in‚0 disk.md0p1.out‚ 645.41015625 disk.sdb2.out‚669.3583984375û€ñJ2±- _wandbruntime‚6135  _runtime‚6135q=SlÒ²ìM K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 142752032 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚224318537201664 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚237162032801792 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚203526544521888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99850780 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142778402 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10403675935640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚224318285516800 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10611220771328 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97587600 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.all€õl ocated‚59673886 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚213929974436876 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10403446954348 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚203526527482528 % _timestamp‚1.7884700827106128e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59673884 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚213930220457528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _step‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚213885281962496 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 242602810 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10610932262400 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚224318537201664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 242602810 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚213885263959552 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚237161720352768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 242629182 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚97588265 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 242629182 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚665 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚226550788090368 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10433255239168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37913716 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 _runtime‚6135.896281862 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99850778 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582í#Àu÷&³ñM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚213885281962496 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 242602810 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10610932262400 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚224318537201664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 242602810 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚213885263959552 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚237161720352768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 242629182 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚97588265 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 242629182 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚665 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚226550788090368 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10433255239168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37913716 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 _runtime‚6135.896281862 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99850778 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 142752032 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚224318537201664 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚237162032801792 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚203526544521888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99850780 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142778402 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10403675935640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚224318285516800 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10611220771328 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97587600 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59673886 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚213929974436876 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10403446954348 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚203526527482528 % _timestamp‚1.7884700827106128e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59673884 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚213930220457528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _step‚430®gZ•2´- _wandbruntime‚6135  _runtime‚6135‡ò¡!ZµÔ 1loss/FBt1/assignment_loss‚0.19505824982657316 1loss/FBt2/assignment_loss‚0.34416696591846024 /loss/FRt3/detection_loss‚1.4414722414940229 _runtime‚6135.897817499 2loss/SRqqt4/assignment_loss‚1.8281904290829385 0loss/FBt4/assignment_loss‚0.5424120722142437 /loss/FBt3/detection_loss‚0.5155671516136415 1loss/SRqqt3/detection_loss‚0.7020378438844568 /loss/FBt4/detection_loss‚2.4689815640449524 epoch‚4 0loss/FRt2/assignment_loss‚3.4710604110184837 .loss/FRt4/detection_loss‚2.751347711009364 %loss/total_loss‚43.47235963633809 % _timestamp‚1.7884700827131195e+09 /loss/FRt1/assignment_loss‚3.526235625550554 2loss/SRqqt1/assignment_loss‚2.1432702953859075 1loss/SRqqt2/assignment_loss‚1.756744434857475 /loss/FRt2/detection_loss‚2.5841067317258637 1loss/SRqqt1/detection_loss‚2.2763412841936437 1loss/SRqqt4/detection_loss‚2.2994826316833494 .loss/FRt3/assignment_loss‚1.42146881642165 .loss/FRt1/detection_loss‚2.687728533068219 1loss/SRqqt2/detection_loss‚2.1266373906816756 /loss/FBt1/detection_loss‚2.2845781594514847 _step‚431 2loss/SRqqt3/assignment_loss‚1.0432894245677051 .loss/FBt2/detection_loss‚2.075077243645986 trainer/global_step‚5249 /loss/FRt4/assignment_loss‚2.901840094597109 1loss/FBt3/assignment_loss‚0.08526433040033293,cÏü_¶Ù 1loss/FBt1/assignment_loss‚0.19505824982657316 1loss/FBt2/assignment_loss‚0.34416696591846024 /loss/FRt3/detection_loss‚1.4414722414940229 _runtime‚6135.897817499 2loss/SRqqt4/assignment_loss‚1.8281904290829385 0loss/FBt4/assignment_loss‚0.5424120722142437 /loss/FBt3/detection_loss‚0.5155671516136415 1loss/SRqqt3/detection_loss‚0.7020378438844568 /loss/FBt4/detection_loss‚2.4689815640449524 epoch‚4 0loss/FRt2/assignment_loss‚3.4710604110184837 .loss/FRt4/detection_loss‚2.751347711009364 %loss/total_loss‚43.47235963633809 % _timestamp‚1.7884700827131195e+09 /loss/FRt1/assignment_loss‚3.526235625550554 2loss/SRqqt1/assignment_loss‚2.1432702953859075 1loss/SRqqt2/assignment_loss‚1.756744434857475 /loss/FRt2/detection_loss‚2.5841067317258637 1loss/SRqqt1/detection_loss‚2.2763412841936437 1loss/SRqqt4/detection_loss‚2.2994826316833494 .loss/FRt3/assignment_loss‚1.42146881642165 .loss/FRt1/detection_loss‚2.687728533068219 1loss/SRqqt2/detection_loss‚2.1266373906816756 /loss/FBt1/detection_loss‚2.2845781594514847 _step‚431 2loss/SRqqt3/assignment_loss‚1.0432894245677051 .loss/FBt2/detection_loss‚2.075077243645986 trainer/global_step‚5249 /loss/FRt4/assignment_loss‚2.901840094597109 1loss/FBt3/assignment_loss‚0.08526433040033293¯h3U^‘·‚(:† ÐÆçÔ®¶àˆ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚136.466' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.1165 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yv4Ö{¸‚(:ð ÐÆçÔ¸œ¡‰ disk.md0.in‚7.125 disk.sda1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 1.62109375! disk.nvme0n1.in‚ 328975.703125 disk.dm-0.in‚529698.98828125 disk.rbd0.in‚0.015625# disk.nvme1n1.out‚45678.47265625 network.sent‚ 1.2922837e+07% disk./.usageGB‚2974.3261909484863 disk.sda1.out‚0! disk.sda2.out‚670.83154296875 disk.md0p1.in‚7.125 disk.sdb.out‚670.83154296875# proc.memory.rssMB‚ 3030.74609375 disk.sdb1.out‚0 disk.sda.out‚670.83154296875" disk.nvme0n1.out‚ 32891.3515625 disk.md0.out‚ 646.828125 disk.rbd0.out‚ 1.68359375* proc.memory.availableMB‚10472.48046875 proc.cpu.threads‚17 disk.sda.in‚ 3.62890625 disk.dm-0.out‚78569.82421875 disk.sda2.in‚ 3.62890625$ memory_percent‚36.08105182647705) proc.memory.percent‚18.49820613861084 cpu‚25.176341188869834! disk.sdb2.out‚670.83154296875# disk.nvme1n1.in‚200723.28515625 network.recv‚ 2.064796e+06) disk./.usagePercent‚83.16340515114054 disk.sdb2.in‚ 3.49609375 disk.sdb.in‚ 3.49609375 disk.sdb1.in‚0 disk.md0p1.out‚ 646.828125‘0R‘¹‚(:† ßÆçÔºÕ݇ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚136.134' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.0335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üœç©wº‚(:ì ßÆçÔ®Ó’ˆ) disk./.usagePercent‚83.16269436535447$ memory_percent‚36.08107566833496 disk.sda2.out‚672.5283203125 disk.rbd0.out‚ 1.68359375 disk.sdb.in‚ 3.49609375# disk.nvme0n1.out‚33177.68359375 disk.sda1.in‚0 disk.sda2.in‚ 3.62890625) proc.memory.percent‚18.49820613861084 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 1.62109375# disk.nvme1n1.in‚200723.28515625 disk.sdb.out‚672.5283203125 cpu‚25.176004546678296 disk.sda.out‚672.5283203125 network.recv‚ 2.070827e+06$ disk./.usageGB‚2974.300769805908 disk.sda1.out‚0 disk.md0p1.out‚ 648.4609375 disk.md0.out‚ 648.4609375 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0p1.in‚7.125 disk.sdb2.out‚672.5283203125 disk.dm-0.out‚78856.25390625) proc.memory.availableMB‚ 10472.4765625 proc.cpu.threads‚17 disk.rbd2.in‚0" disk.nvme1n1.out‚ 45678.5703125 disk.dm-0.in‚529698.98828125# proc.memory.rssMB‚ 3030.74609375 network.sent‚ 1.3021402e+07 disk.sdb2.in‚ 3.49609375 disk.md0.in‚7.125 disk.sda.in‚ 3.62890625! disk.nvme0n1.in‚ 328975.703125"•à컂(:… îÆçÔäï… gpu.0.gpu‚10.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚112.572' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.143 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹«gëu¼‚(:ê îÆçÔ…¤ˆ disk.sdb1.out‚0 disk.dm-0.in‚529699.05859375 network.recv‚ 2.075603e+06 disk.sdb2.out‚674.5869140625 disk.rbd2.out‚ 1.62109375# disk.nvme1n1.out‚45679.03515625) proc.memory.percent‚18.49820613861084 disk.dm-0.out‚78857.51953125 disk.sda2.in‚ 3.62890625# memory_percent‚36.0811710357666 disk.md0p1.out‚ 650.453125 disk.sda.out‚674.5869140625 cpu‚25.175618745253733 proc.cpu.threads‚17 disk.md0p1.in‚7.125 disk.sdb2.in‚ 3.49609375# disk.nvme1n1.in‚200723.28515625 disk.sdb1.in‚0" disk.nvme0n1.in‚328975.7734375 disk.sda1.out‚0) disk./.usagePercent‚83.16146414277699% disk./.usageGB‚2974.2567710876465 disk.rbd0.out‚ 1.68359375 disk.sdb.out‚674.5869140625 disk.sda2.out‚674.5869140625 disk.rbd0.in‚0.015625 disk.sdb.in‚ 3.49609375 disk.sda.in‚ 3.62890625 network.sent‚ 1.3025752e+07! disk.nvme0n1.out‚ 33187.734375) proc.memory.availableMB‚ 10472.4609375 disk.md0.in‚7.125 disk.md0.out‚ 650.453125 disk.rbd2.in‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 3030.74609375«”¼É2½- _wandbruntime‚6191  _runtime‚6191ë YƾÀ & lr-AdamW/pg1‚0.0005689195795151845 & lr-AdamW/pg2‚0.0005689195795151845 trainer/global_step‚5299 % _timestamp‚1.7884701379213207e+09 _runtime‚ 6191.10636097 _step‚432 Ý›ÇË¿Å & lr-AdamW/pg1‚0.0005689195795151845 & lr-AdamW/pg2‚0.0005689195795151845 trainer/global_step‚5299 % _timestamp‚1.7884701379213207e+09 _runtime‚ 6191.10636097 _step‚432°Øy-Ê2À- _wandbruntime‚6191  _runtime‚6191Oõ@€ÁåO M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 100746253 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚226339553010176 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚215810297775104 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10529273238016 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚215810279772160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10529506906624 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 144089636 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 144089636 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38251340 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10709139753984 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 244809545 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10529273238016 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚5299 _step‚433 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚226339804681728 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38250700 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚215864941502174 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100746253 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10499426505870 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 244835891 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144063292 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60203184 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚226339553010176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚239302160145920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚228593326562816 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚228593302622720 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608ΊРð'ÂêO XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10529273238016 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚5299 _step‚433 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚226339804681728 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38250700 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚215864941502174 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100746253 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10499426505870 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 244835891 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144063292 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60203184 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚226339553010176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚239302160145920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚228593326562816 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚228593302622720 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 100746253 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚226339553010176 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚215810297775104 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10529273238016 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚215810279772160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10529506906624 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 144089636 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 144089636 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38251340 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10709139753984 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 244809545 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104±Ü¶2Ã- _wandbruntime‚6192  _runtime‚6192.rK$õ&ÄïM ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚98471786 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚226381054757888 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244880884 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 100764528 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10710849624064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 100764530 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38258473 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 % _timestamp‚1.7884701389544432e+09 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60213986 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚239345836216832 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚215849565809152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 244880884 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 _runtime‚ 6192.14007737 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100764530 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10501384994856 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚215849583812096 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚673 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚215849565809152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚215904676369236 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚215849583812096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10531470945792 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚5299 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚226381054757888 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 144089984 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38257802 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10531470945792 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚228634986592768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚205403062393088 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736Uµùÿ[#ÅôM SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 100764530 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38258473 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 % _timestamp‚1.7884701389544432e+09 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60213986 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚239345836216832 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚215849565809152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 244880884 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 _runtime‚ 6192.14007737 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100764530 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10501384994856 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚215849583812096 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚673 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚215849565809152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚215904676369236 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚215849583812096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10531470945792 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚5299 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚226381054757888 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 144089984 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38257802 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10531470945792 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚228634986592768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚205403062393088 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚98471786 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 WIDeviceStatsMonitor.on_train_baãA8Ÿtch_end/reserved_bytes.small_pool.allocated‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚226381054757888 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244880884 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 100764528 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10710849624064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608²@‹¿i2Æ- _wandbruntime‚6192  _runtime‚6192Œ±þbÇÜ 0loss/FRt3/assignment_loss‚1.4962968627611797 0loss/FBt2/assignment_loss‚0.1326085126900125 /loss/FBt2/detection_loss‚2.3827445805072784 trainer/global_step‚5299 /loss/FRt2/detection_loss‚2.1566259102501837 /loss/FRt3/detection_loss‚1.4900433763861656 1loss/SRqqt1/detection_loss‚1.8022882392009099 1loss/SRqqt3/detection_loss‚1.0486321042798743 /loss/FBt3/detection_loss‚0.6136509399126533 0loss/FRt2/assignment_loss‚3.1878202664546476 2loss/FBt4/assignment_loss‚0.026303766795232757 /loss/FRt1/detection_loss‚2.4807966933017824 0loss/SRqqt2/detection_loss‚2.363420197457978 2loss/SRqqt2/assignment_loss‚1.3751818679950454 .loss/FRt4/detection_loss‚2.802058145955757 epoch‚4 0loss/FRt4/assignment_loss‚3.8235956165525646 2loss/SRqqt3/assignment_loss‚1.0017930454798514 2loss/FBt1/assignment_loss‚0.006799179493029328 1loss/SRqqt4/detection_loss‚2.4214783078059554 _step‚435 1loss/FBt3/assignment_loss‚0.03176534291281866 % _timestamp‚1.7884701389566393e+09 2loss/SRqqt1/assignment_loss‚0.9076697740781431 2loss/SRqqt4/assignment_loss‚1.6113145693670958 /loss/FBt1/detection_loss‚2.8443451151251793 /loss/FBt4/detection_loss‚2.5967916548252106 %loss/total_loss‚40.94863272392478 _runtime‚6192.141406646 0loss/FRt1/assignment_loss‚2.3446086543362314 7ôgÈá 0loss/FRt2/assignment_loss‚3.1878202664546476 2loss/FBt4/assignment_loss‚0.026303766795232757 /loss/FRt1/detection_loss‚2.4807966933017824 0loss/SRqqt2/detection_loss‚2.363420197457978 2loss/SRqqt2/assignment_loss‚1.3751818679950454 .loss/FRt4/detection_loss‚2.802058145955757 epoch‚4 0loss/FRt4/assignment_loss‚3.8235956165525646 2loss/SRqqt3/assignment_loss‚1.0017930454798514 2loss/FBt1/assignment_loss‚0.006799179493029328 1loss/SRqqt4/detection_loss‚2.4214783078059554 _step‚435 1loss/FBt3/assignment_loss‚0.03176534291281866 % _timestamp‚1.7884701389566393e+09 2loss/SRqqt1/assignment_loss‚0.9076697740781431 2loss/SRqqt4/assignment_loss‚1.6113145693670958 /loss/FBt1/detection_loss‚2.8443451151251793 /loss/FBt4/detection_loss‚2.5967916548252106 %loss/total_loss‚40.94863272392478 _runtime‚6192.141406646 0loss/FRt1/assignment_loss‚2.3446086543362314 0loss/FRt3/assignment_loss‚1.4962968627611797 0loss/FBt2/assignment_loss‚0.1326085126900125 /loss/FBt2/detection_loss‚2.3827445805072784 trainer/global_step‚5299 /loss/FRt2/detection_loss‚2.1566259102501837 /loss/FRt3/detection_loss‚1.4900433763861656 1loss/SRqqt1/detection_loss‚1.8022882392009099 1loss/SRqqt3/detection_loss‚1.0486321042798743 /loss/FBt3/detection_loss‚0.6136509399126533³nΗ{œÉ‚(:‘ ýÆçÔЫ²ˆ gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚110.681' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚27.670250000000003 gpu.0.smClock‚1245 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìÉ«ósÊ‚(:è ýÆçÔ ®ž‰ disk.sdb1.out‚0 disk.rbd2.out‚ 1.62109375 disk.sdb.in‚ 3.49609375 disk.sdb.out‚ 675.908203125* proc.memory.availableMB‚10472.23046875 network.recv‚ 2.079888e+06 disk.sda.in‚ 3.62890625 disk.sda.out‚ 675.908203125 disk.nvme0n1.out‚ 33446.46875 disk.dm-0.in‚529699.2265625 disk.md0p1.in‚7.125) proc.memory.percent‚18.49820613861084 proc.cpu.threads‚17% disk./.usageGB‚2974.8278846740723 disk.rbd0.out‚ 1.68359375 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda2.in‚ 3.62890625 disk.sda2.out‚ 675.908203125 network.sent‚ 1.3032191e+07# disk.nvme0n1.in‚328975.80078125# memory_percent‚36.0825777053833# proc.memory.rssMB‚ 3030.74609375 disk.sdb1.in‚0 disk.dm-0.out‚ 79125.5859375 cpu‚25.175308165242303 disk.rbd0.in‚0.015625# disk.nvme1n1.in‚200723.42578125 disk.md0.in‚7.125" disk.nvme1n1.out‚ 45679.1171875 disk.sda1.out‚0 disk.md0p1.out‚ 651.69921875 disk.sdb2.in‚ 3.49609375 disk.sdb2.out‚ 675.908203125) disk./.usagePercent‚83.17743271768305 disk.md0.out‚ 651.699218750°&ä‘Ë‚(:† ŒÇçÔۢ݇ gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚140.478' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.1195 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ°6mÌ‚(:â ŒÇçÔ—µ”ˆ disk.sdb1.in‚0) disk./.usagePercent‚83.18086676017525% disk./.usageGB‚2974.9507026672363 disk.sda1.out‚0 disk.sdb2.in‚ 3.49609375 disk.md0.out‚653.25 disk.sda1.in‚0 disk.sda2.in‚ 3.62890625 disk.rbd2.in‚0 disk.sdb.in‚ 3.49609375 cpu‚25.174936055998067 disk.sda2.out‚ 677.517578125 disk.rbd2.out‚ 1.62109375) proc.memory.percent‚18.49820613861084" disk.nvme0n1.out‚ 33705.8203125 disk.rbd0.out‚ 1.68359375 disk.sda.out‚ 677.517578125% memory_percent‚36.083102226257324 proc.cpu.threads‚17 disk.sdb1.out‚0 network.sent‚ 1.3141305e+07# disk.nvme0n1.in‚328975.80078125# disk.nvme1n1.in‚200723.42578125 disk.dm-0.in‚529699.2265625 disk.md0.in‚7.125 disk.md0p1.out‚653.25 network.recv‚ 2.086455e+06 disk.dm-0.out‚79385.33984375 disk.rbd0.in‚0.015625# disk.nvme1n1.out‚45679.51953125 disk.sda.in‚ 3.62890625 disk.md0p1.in‚7.125 disk.sdb2.out‚ 677.517578125 disk.sdb.out‚ 677.517578125* proc.memory.availableMB‚10472.14453125# proc.memory.rssMB‚ 3030.74609375/ÚìÔ‘Í‚(:† ›ÇçÔ¼û—‡ gpu.0.gpu‚43.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚138.302' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.5755 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒÒ„¥t΂(:é ›ÇçÔÜŸˆˆ disk.sdb.out‚ 679.212890625 disk.sda2.out‚ 679.216796875) proc.memory.percent‚18.49820613861084 proc.cpu.threads‚17 network.sent‚ 1.3146628e+07# disk.nvme1n1.in‚200723.42578125 disk.sdb.in‚ 3.49609375 disk.sda.in‚ 3.62890625# disk.nvme0n1.in‚328975.80078125# disk.nvme0n1.out‚33908.43359375$ disk./.usageGB‚2975.075511932373 disk.md0.out‚ 654.890625 disk.rbd0.in‚0.015625 disk.sda2.in‚ 3.62890625# proc.memory.rssMB‚ 3030.74609375 cpu‚25.174597492689944 network.recv‚ 2.090383e+06) disk./.usagePercent‚83.18435647946521 disk.rbd2.out‚ 1.62109375" disk.nvme1n1.out‚ 45679.5859375 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.dm-0.out‚79588.01953125 disk.md0p1.in‚7.125 disk.sdb2.in‚ 3.49609375 disk.md0p1.out‚ 654.89453125 disk.rbd0.out‚ 1.68359375 disk.sda1.in‚0 disk.sda1.out‚0% memory_percent‚36.081600189208984 disk.rbd2.in‚0 disk.sda.out‚ 679.212890625 disk.dm-0.in‚529699.2265625( proc.memory.availableMB‚ 10472.390625 disk.sdb2.out‚ 679.212890625 disk.md0.in‚7.125;Q3‘Ï‚(:† ªÇçÔÄ¢ö‡ gpu.0.gpu‚7.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚139.251' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.81275 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡J0wЂ(:ì ªÇçÔÑÞ¿ˆ network.recv‚ 2.095051e+06 disk.sdb1.in‚0 disk.dm-0.in‚ 529708.53125 disk.rbd0.out‚ 1.68359375 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.availableMB‚10472.40234375 disk.rbd0.in‚0.015625 disk.sdb.in‚ 3.49609375 disk.sda2.in‚ 3.62890625 disk.md0.in‚7.125 disk.md0.out‚ 656.4453125 disk.rbd2.in‚0! disk.nvme0n1.out‚ 34171.796875 disk.md0p1.in‚7.125 proc.cpu.threads‚17 network.sent‚ 1.3150745e+07! disk.sdb2.out‚680.81884765625 disk.sda.in‚ 3.62890625# proc.memory.rssMB‚ 3030.74609375 disk.sdb2.in‚ 3.49609375" disk.nvme1n1.out‚ 45679.7890625 disk.sdb.out‚680.81884765625! disk.sda2.out‚680.81884765625 disk.md0p1.out‚ 656.4453125 cpu‚25.174296384586444) disk./.usagePercent‚83.18434048038539 disk.rbd2.out‚ 1.62109375" disk.nvme1n1.in‚200727.9453125% memory_percent‚36.081528663635254$ disk./.usageGB‚2975.074939727783 disk.sdb1.out‚0 disk.sda.out‚680.81884765625" disk.nvme0n1.in‚328980.5859375 disk.dm-0.out‚ 79851.5859375) proc.memory.percent‚18.49820613861084Z«„2Ñ- _wandbruntime‚6246  _runtime‚6246ZXQÆÒÀ & lr-AdamW/pg1‚0.0005633991685784581 & lr-AdamW/pg2‚0.0005633991685784581 trainer/global_step‚5349 % _timestamp‚1.7884701932745316e+09 _runtime‚ 6246.45949838 _step‚436Þ‰RËÓÅ _step‚436 & lr-AdamW/pg1‚0.0005633991685784581 & lr-AdamW/pg2‚0.0005633991685784581 trainer/global_step‚5349 % _timestamp‚1.7884701932745316e+09 _runtime‚ 6246.45949838´Ã[2Ô- _wandbruntime‚6246  _runtime‚6246½˜á„ë'ÕåO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚692 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚207242049906864 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 145427588 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚217774581621760 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10627488944640 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚241486568033792 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚217774581621760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.788470193275744e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10627722613248 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101660005 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚217839643434326 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚217774599624704 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚217839397413882 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚228402070566400 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101660003 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38596998 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 145401244 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚217774599624704 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10627488944640 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 145401244 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _runtime‚6246.460869827 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101660005 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10627722613248 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚228402070566400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38596306 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101660003 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚99340282 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚437 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 145427588 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10808758640640 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚207242032867504 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10597364546378 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 247087593 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚241486259765760 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚692 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚207242049906864 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 145427588 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚217774581621760 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10627488944640 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚241486568033792 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚217774581621760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.788470193275744e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10627722613248 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101660005 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚217839643434326 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚217774599624704 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚217839397413882 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚228402070566400 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101660003 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38596998 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 145401244 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚217774599624704 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10627488944640 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 145401244 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _runtime‚6246.460869827 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101660005 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10627722613248 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚228402070566400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38596306 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101660003 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚99340282 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚437 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 145427588 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10808758640640 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚207242032867504 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10597364546378 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 247087593 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚241486259765760 B¾ØðM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚230719185095168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101678278 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38603657 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚217813867658752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38602988 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 D>"¾È8 DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101678280 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10811058061312 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚217878886260292 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145454306 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10629686652416 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚217813885661696 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚230719209035264 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 278023168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 145427936 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚228443572314112 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10629686652416 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚669 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 247106214 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚228443320629248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60754084 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 247132586 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99357072 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301963264 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10599323035364 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚217813885661696 trainer/global_step‚5349 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101678278 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚228443572314112 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚241530267096576 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 247132586 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚207279563224928 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10599552016656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10629452970496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚671 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10810780038144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 145427936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚438 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚6247.466175688 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101678280 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10811058061312 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚217878886260292 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145454306 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10629686652416 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚217813885661696 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚230719209035264 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 278023168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 145427936 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚228443572314112 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10629686652416 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚669 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 247106214 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚228443320629248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60754084 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 247132586 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99357072 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301963264 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10599323035364 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚217813885661696 trainer/global_step‚5349 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101678278 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚228443572314112 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚241530267096576 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 247132586 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚207279563224928 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10599552016656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10629452970496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚671 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10810780038144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 145427936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚438 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0¶;Kµ}2Ú- _wandbruntime‚6247  _runtime‚6247éU¢öWÛÑ 1loss/SRqqt2/assignment_loss‚1.956894837319851 0loss/SRqqt3/detection_loss‚0.948362509382295 %loss/total_loss‚44.04523826910712 _runtime‚6247.467900286 /loss/FRt4/detection_loss‚3.3907980978488923 epoch‚4 _step‚439 0loss/FRt1/assignment_loss‚3.0661417841911316 /loss/FRt3/assignment_loss‚1.353531059147655 .loss/FRt3/detection_loss‚1.575147599197816 /loss/FBt4/detection_loss‚2.1858266847474233 2loss/SRqqt4/assignment_loss‚1.5758776583243161 /loss/FRt1/detection_loss‚2.7705962931787647 1loss/SRqqt1/detection_loss‚1.7931471776503782 -loss/FBt1/detection_loss‚2.04712667564551 /loss/FRt2/assignment_loss‚3.434799905093211 1loss/FBt1/assignment_loss‚0.10886250306696373 1loss/FBt2/assignment_loss‚0.15458815731593134 /loss/FBt2/detection_loss‚3.4258206288019815 $ _timestamp‚1.788470194283162e+09 1loss/SRqqt1/assignment_loss‚1.467379492922471 2loss/FBt3/assignment_loss‚0.057672081293643436 /loss/FRt2/detection_loss‚1.8780431194046407 0loss/SRqqt2/detection_loss‚2.325161311775446 /loss/SRqqt4/detection_loss‚2.46243148855865 trainer/global_step‚5349 0loss/FRt4/assignment_loss‚3.6294740617275236 2loss/SRqqt3/assignment_loss‚1.1324196979812868 0loss/FBt4/assignment_loss‚0.3911558006357154 /loss/FBt3/detection_loss‚0.9139796438956119抜/\ÜÖ 0loss/FRt4/assignment_loss‚3.6294740617275236 2loss/SRqqt3/assignment_loss‚1.1324196979812868 0loss/FBt4/assignment_loss‚0.3911558006357154 /loss/FBt3/detection_loss‚0.9139796438956119 1loss/SRqqt2/assignment_loss‚1.956894837319851 0loss/SRqqt3/detection_loss‚0.948362509382295 %loss/total_loss‚44.04523826910712 _runtime‚6247.467900286 /loss/FRt4/detection_loss‚3.3907980978488923 epoch‚4 _step‚439 0loss/FRt1/assignment_loss‚3.0661417841911316 /loss/FRt3/assignment_loss‚1.353531059147655 .loss/FRt3/detection_loss‚1.575147599197816 /loss/FBt4/detection_loss‚2.1858266847474233 2loss/SRqqt4/assignment_loss‚1.5758776583243161 /loss/FRt1/detection_loss‚2.7705962931787647 1loss/SRqqt1/detection_loss‚1.7931471776503782 -loss/FBt1/detection_loss‚2.04712667564551 /loss/FRt2/assignment_loss‚3.434799905093211 1loss/FBt1/assignment_loss‚0.10886250306696373 1loss/FBt2/assignment_loss‚0.15458815731593134 /loss/FBt2/detection_loss‚3.4258206288019815 $ _timestamp‚1.788470194283162e+09 1loss/SRqqt1/assignment_loss‚1.467379492922471 2loss/FBt3/assignment_loss‚0.057672081293643436 /loss/FRt2/detection_loss‚1.8780431194046407 0loss/SRqqt2/detection_loss‚2.325161311775446 /loss/SRqqt4/detection_loss‚2.46243148855865 trainer/global_step‚5349·ÎøÑEœÝ‚(:‘ ¹ÇçÔ¨†½‡ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚126.497' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚31.624249999999996 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ð”aVyÞ‚(:î ¹ÇçԹƄ‰ disk.sdb2.in‚ 3.49609375 disk.rbd2.in‚0 disk.dm-0.in‚529713.6953125 disk.sda1.out‚0 disk.md0.out‚ 657.9921875 disk.sdb.out‚682.43310546875 disk.sdb1.in‚0) proc.memory.percent‚18.49820613861084 cpu‚25.173875664748415 disk.md0p1.in‚7.125 disk.md0.in‚7.125 disk.rbd0.out‚ 1.68359375 disk.nvme0n1.in‚ 328983.21875* proc.memory.availableMB‚10472.36328125 disk.md0p1.out‚ 657.9921875# disk.nvme1n1.out‚45679.89453125 disk.sdb.in‚ 3.49609375 disk.sda1.in‚0% memory_percent‚36.081767082214355 network.sent‚ 1.3169399e+07 disk.rbd0.in‚0.015625 disk.sda.in‚ 3.62890625! disk.sda2.out‚682.43310546875 network.recv‚ 2.099361e+06) disk./.usagePercent‚83.17668769386619$ disk./.usageGB‚2974.801239013672 disk.rbd2.out‚ 1.62109375 disk.sda.out‚682.43310546875" disk.nvme1n1.in‚200730.4765625! disk.nvme0n1.out‚ 34432.265625 disk.dm-0.out‚80112.16015625 disk.sda2.in‚ 3.62890625! disk.sdb2.out‚682.43310546875 disk.sdb1.out‚0# proc.memory.rssMB‚ 3030.74609375 proc.cpu.threads‚17÷]R’ß‚(:‡ ÈÇçÔ¶“ä… gpu.0.gpu‚43.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚124.661' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.16525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ž jrà‚(:ç ÈÇçÔŠŽç† disk.dm-0.out‚80224.01171875 disk.sdb2.in‚ 3.49609375" disk.nvme1n1.in‚200730.4765625% disk./.usageGB‚2974.7569885253906 disk.sda.in‚ 3.62890625 disk.nvme0n1.in‚ 328983.21875 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0 disk.sdb.in‚ 3.49609375* proc.memory.availableMB‚10472.11328125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda.out‚684.0654296875 disk.nvme0n1.out‚34543.75 disk.md0.in‚7.125% memory_percent‚36.083292961120605# proc.memory.rssMB‚ 3030.74609375) proc.memory.percent‚18.49820613861084 disk.sda2.out‚684.0654296875 disk.sdb2.out‚684.0654296875 cpu‚25.173480528241647 network.recv‚ 2.105724e+06 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 1.68359375 disk.sdb.out‚684.0654296875# disk.nvme1n1.out‚45680.26171875) disk./.usagePercent‚83.17545043169358 disk.rbd2.out‚ 1.62109375 proc.cpu.threads‚17 network.sent‚ 1.3264292e+07 disk.md0p1.in‚7.125 disk.md0p1.out‚ 659.5703125 disk.md0.out‚ 659.5703125 disk.dm-0.in‚529713.6953125 disk.sda2.in‚ 3.62890625•/µ qá‚(:æ ×ÇçÔ¼¿‚‡# disk.nvme1n1.out‚45680.32421875 proc.cpu.threads‚17 network.sent‚ 1.326967e+07 disk.sdb.out‚686.01806640625 disk.sda.out‚686.01806640625 disk.sda2.in‚ 3.62890625 disk.md0p1.out‚ 661.46875 disk.sdb2.in‚ 3.49609375 disk.rbd0.in‚0.015625 disk.sdb.in‚ 3.49609375 disk.sda.in‚ 3.62890625# disk.nvme0n1.out‚34743.73046875 disk.dm-0.in‚529713.6953125 disk.sda1.in‚0) disk./.usagePercent‚83.17732211071124" disk.nvme1n1.in‚200730.4765625 disk.nvme0n1.in‚ 328983.21875 disk.md0.in‚7.125 disk.md0.out‚ 661.46875 cpu‚25.17310647239755 disk.rbd2.out‚ 1.62109375 disk.sdb1.in‚0# proc.memory.rssMB‚ 3030.74609375$ disk./.usageGB‚2974.823928833008 disk.md0p1.in‚7.125( proc.memory.availableMB‚ 10472.359375 disk.rbd0.out‚ 1.68359375 disk.dm-0.out‚ 80424.0546875 disk.sda1.out‚0! disk.sda2.out‚686.01806640625% memory_percent‚36.081790924072266 network.recv‚ 2.10973e+06 disk.rbd2.in‚0 disk.sdb1.out‚0! disk.sdb2.out‚686.01806640625) proc.memory.percent‚18.49820613861084Å"’â‚(:‡ ×ÇçÔÆÅƉ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚137.969' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.49225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Âx’ã‚(:‡ æÇçÔã³û… gpu.0.gpu‚24.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚144.821' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.20525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶0>:yä‚(:î æÇçÔÆàć disk.md0p1.in‚7.125" disk.nvme1n1.in‚200730.4765625# proc.memory.rssMB‚ 3030.74609375 disk.md0.out‚ 662.8203125 cpu‚25.17273174143344 network.recv‚ 2.114313e+06 disk.rbd2.in‚0) disk./.usagePercent‚83.18088286591559 disk.rbd2.out‚ 1.62109375 proc.cpu.threads‚17% disk./.usageGB‚2974.9512786865234 disk.sdb.out‚687.42529296875 disk.sda.in‚ 3.62890625 disk.sda1.out‚0 disk.md0p1.out‚ 662.8203125 disk.md0.in‚7.125# disk.nvme1n1.out‚45680.53515625$ memory_percent‚36.08248233795166" disk.nvme0n1.out‚ 34970.8828125 disk.sda1.in‚0! disk.sda2.out‚687.42529296875! disk.sdb2.out‚687.42529296875* proc.memory.availableMB‚10472.24609375 network.sent‚ 1.3275993e+07 disk.sda.out‚687.42529296875 disk.dm-0.in‚529713.6953125 disk.sdb.in‚ 3.49609375 disk.sdb1.in‚0 disk.sda2.in‚ 3.62890625 disk.sdb2.in‚ 3.49609375 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 1.68359375) proc.memory.percent‚18.49820613861084 disk.sdb1.out‚0 disk.nvme0n1.in‚ 328983.21875 disk.dm-0.out‚80651.41796875`yÖ«2å- _wandbruntime‚6300  _runtime‚63003M!*Åæ¿ & lr-AdamW/pg1‚0.0005578472534531463 & lr-AdamW/pg2‚0.0005578472534531463 trainer/global_step‚5399 $ _timestamp‚1.788470247575185e+09 _runtime‚ 6300.76019552 _step‚440™ÇÊçÄ & lr-AdamW/pg1‚0.0005578472534531463 & lr-AdamW/pg2‚0.0005578472534531463 trainer/global_step‚5399 $ _timestamp‚1.788470247575185e+09 _runtime‚ 6300.76019552 _step‚440¸âšÕ2è- _wandbruntime‚6300  _runtime‚6300~ÝÈçPéèO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚219738883471360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚219814099346034 % _timestamp‚1.7884702475760827e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚219738883471360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚232761699627520 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10909049186304 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 100228207 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 146739196 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38944823 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚719 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚243670772753920 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10695531567970 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61283382 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0E»}”ž JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61283384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102573753 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚209118550738704 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚243670466583040 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10908766955520 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚230464588122624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38944104 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚209118567778064 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 146765540 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 146739196 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 102573755 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚441 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚230464588122624 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚232761723567616 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚219813853325590 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10725704651264 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 100227486 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282230784 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 249312949 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚230464839794176 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102573753 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚721 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306170880 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10725938319872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 249339295 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 trainer/global_step‚5399¢áó'êíO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 249312949 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚230464839794176 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102573753 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚721 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306170880 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10725938319872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 249339295 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 trainer/global_step‚5399 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚219738883471360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚219814099346034 % _timestamp‚1.7884702475760827e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚219738883471360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚232761699627520 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10909049186304 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 100228207 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 146739196 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38944823 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚719 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚243670772753920 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10695531567970 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61283382 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61283384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102573753 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚209118550738704 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚243670466583040 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10908766955520 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚230464588122624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38944104 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚209118567778064 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 146765540 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 146739196 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 102573755 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚441 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚230464588122624 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚232761723567616 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚219813853325590 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10725704651264 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 100227486 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282230784¹Fñc2ë- _wandbruntime‚6301  _runtime‚6301eçß÷&ìñM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788470248543655e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102592030 _step‚442 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚209156098135488 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 249384288 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚209156081096128 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102592030 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10727668677120 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 100245841 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚243714453791232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 249357916 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10727902359040 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚243714141342208 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 249384288 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚230506089870336 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 146765888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10910757744640 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 249357916 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚219853588192652 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100245157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 146792258 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102592028 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚230505838185472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10911046253568 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0öÿ®¼#íöM XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚219778169508352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38950973 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61294186 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102592028 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚219778187511296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚209156098135488 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 249384288 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚209156081096128 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102592030 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10727668677120 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 100245841 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚243714453791232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 249357916 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10727902359040 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚243714141342208 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 249384288 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚230506089870336 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 146765888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10910757744640 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 249357916 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚219853588192652 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100245157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 146792258 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102592028 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚230505838185472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10911046253568 GADeviceStatsMonitor.on_train_batch_end/ove·-¬ÓÙ rsize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788470248543655e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102592030 _step‚442 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚221703203323904 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304073728 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39288574 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚210995068609904 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10793240627394 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 251590997 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚234845898129920 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 103487505 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚232527357350400 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103487503 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 148103492 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚221703203323904 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚245854548056064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚245854852129792 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚232527105678848 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61823484 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚210995085649264 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚445 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚232527357350400 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10824154026496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚221788555257742 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 251564651 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚221703185320960 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚221703185320960 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309çC=ò'üìO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39288574 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚210995068609904 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10793240627394 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 251590997 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚234845898129920 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 103487505 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚232527357350400 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103487503 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 148103492 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚221703203323904 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚245854548056064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚245854852129792 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚232527105678848 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61823484 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚210995085649264 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚445 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚232527357350400 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10824154026496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚221788555257742 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 251564651 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚221703185320960 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚221703185320960 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚221703203323904 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304073728 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496½ìF2ý- _wandbruntime‚6352  _runtime‚6352–‡€e·þòM FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚211032616006688 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103505778 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 103505780 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 251635990 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 251609618 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39294663 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚234887606040064 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚221742489360896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚245898259338240 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚211032598967328 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10826118065664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 148130210 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚232568355741696 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 251609618 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61834286 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11010677238272 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10825884383744 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 % _timestamp‚1.7884702995590591e+09 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚221742489360896 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 103505780 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚221827798083708 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 _runtime‚6352.744716256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 251635990 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚221742471357952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚699 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10825884383744 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10826118065664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚234887582099968 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10795199116380 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101128947 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10795428097672 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚221828044104360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528¯k«_ý&ÿ÷M _step‚446 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 251635990 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 251609618 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39294663 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚234887606040064 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚221742489360896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚245898259338240 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚211032598967328 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10826118065664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 148130210 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚232568355741696 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 251609618 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61834286 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11010677238272 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10825884383744 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 % _timestamp‚1.7884702995590591e+09 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚221742489360896 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 103505780 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚221827798083708 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 _runtime‚6352.744716256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 251635990 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚221742471357952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚699 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10825884383744 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10826118065664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚234887582099968 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10795199116380 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101128947 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10795428097672 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚221828044104360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚211032616006688 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103505778 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 103505780 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F[Õ % _timestamp‚1.7884702995620348e+09 1loss/SRqqt4/assignment_loss‚1.202710278445121 1loss/FBt1/assignment_loss‚0.03259797269544963 /loss/FRt2/detection_loss‚2.4760257093973146 /loss/FBt1/detection_loss‚2.8233275839260648 epoch‚4 /loss/FRt1/assignment_loss‚3.399892498765673 2loss/SRqqt2/assignment_loss‚1.5186019998330336 1loss/FBt3/assignment_loss‚0.06563564669930254 0loss/FRt2/assignment_loss‚3.5249519265360303 1loss/SRqqt2/detection_loss‚2.2498708084607735 1loss/SRqqt4/detection_loss‚2.2183805406093597 -loss/FBt2/detection_loss‚2.47900647521019 _runtime‚6352.746742566 /loss/FBt3/detection_loss‚0.6954458756249791 /loss/FRt4/assignment_loss‚3.612100785970688 .loss/FRt1/detection_loss‚2.856779248373849 1loss/SRqqt3/detection_loss‚0.7421204985941158 1loss/SRqqt1/detection_loss‚2.1556821204721928 .loss/FBt4/detection_loss‚2.292689090425318 %loss/total_loss‚42.41490570089847 trainer/global_step‚5449 0loss/FRt3/assignment_loss‚1.4149556982685143 2loss/SRqqt1/assignment_loss‚1.2393175052478909 0loss/FBt2/assignment_loss‚0.0716768724960275 .loss/FRt3/detection_loss‚1.280245623249613 2loss/SRqqt3/assignment_loss‚0.9138428794548792 /loss/FRt4/detection_loss‚3.1183558881282805 _step‚447 2loss/FBt4/assignment_loss‚0.030692174013810556óŽ)N`‚Ú 2loss/FBt4/assignment_loss‚0.030692174013810556 % _timestamp‚1.7884702995620348e+09 1loss/SRqqt4/assignment_loss‚1.202710278445121 1loss/FBt1/assignment_loss‚0.03259797269544963 /loss/FRt2/detection_loss‚2.4760257093973146 /loss/FBt1/detection_loss‚2.8233275839260648 epoch‚4 /loss/FRt1/assignment_loss‚3.399892498765673 2loss/SRqqt2/assignment_loss‚1.5186019998330336 1loss/FBt3/assignment_loss‚0.06563564669930254 0loss/FRt2/assignment_loss‚3.5249519265360303 1loss/SRqqt2/detection_loss‚2.2498708084607735 1loss/SRqqt4/detection_loss‚2.2183805406093597 -loss/FBt2/detection_loss‚2.47900647521019 _runtime‚6352.746742566 /loss/FBt3/detection_loss‚0.6954458756249791 /loss/FRt4/assignment_loss‚3.612100785970688 .loss/FRt1/detection_loss‚2.856779248373849 1loss/SRqqt3/detection_loss‚0.7421204985941158 1loss/SRqqt1/detection_loss‚2.1556821204721928 .loss/FBt4/detection_loss‚2.292689090425318 %loss/total_loss‚42.41490570089847 trainer/global_step‚5449 0loss/FRt3/assignment_loss‚1.4149556982685143 2loss/SRqqt1/assignment_loss‚1.2393175052478909 0loss/FBt2/assignment_loss‚0.0716768724960275 .loss/FRt3/detection_loss‚1.280245623249613 2loss/SRqqt3/assignment_loss‚0.9138428794548792 /loss/FRt4/detection_loss‚3.1183558881282805 _step‚447¿o]ÉJ‘ƒ‚(:† ¢ÈçÔÇÁ߈ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚136.134' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.0335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åaöj„‚(:ß ¢ÈçÔ’¥˜Š* proc.memory.availableMB‚10472.12890625 disk.sda.in‚ 3.62890625 disk.sda1.out‚0# proc.memory.rssMB‚ 3030.74609375) proc.memory.percent‚18.49820613861084 disk.sda.out‚694.4658203125 network.sent‚ 1.3400127e+07 disk.rbd2.out‚ 1.62109375 disk.sdb1.in‚0 disk.md0.in‚7.125 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.sdb.in‚ 3.49609375 disk.dm-0.out‚ 81389.015625 disk.md0p1.out‚ 669.640625 disk.sdb2.in‚ 3.49609375 disk.sda1.in‚0 disk.nvme0n1.in‚ 328983.21875! disk.nvme1n1.out‚ 45681.515625 network.recv‚ 2.132216e+06 disk.rbd0.out‚ 1.68359375" disk.nvme1n1.in‚200730.4765625 disk.sdb.out‚694.4658203125% memory_percent‚36.083197593688965 cpu‚25.171167468028855( disk./.usagePercent‚83.1471109416299 disk.md0.out‚ 669.640625 disk.rbd0.in‚0.015625 disk.sda2.in‚ 3.62890625 disk.md0p1.in‚7.125 proc.cpu.threads‚17% disk./.usageGB‚2973.7434310913086 disk.sdb2.out‚694.4658203125 disk.nvme0n1.out‚35707.5 disk.dm-0.in‚529713.6953125 disk.sda2.out‚694.4658203125D"Îø‘…‚(:† ±ÈçÔÒÄ͇ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.966' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.9915 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³Ftij†‚(:ß ±ÈçÔÊÒ¥ˆ% disk./.usageGB‚2973.6507987976074 disk.nvme1n1.out‚ 45681.9375 disk.dm-0.in‚529713.6953125 disk.sda1.in‚0 disk.md0p1.out‚ 670.9765625 disk.md0.out‚ 670.9765625 disk.rbd0.out‚ 1.68359375* proc.memory.availableMB‚10471.72265625" disk.nvme1n1.in‚200730.4765625 disk.sdb1.out‚0 disk.sda2.in‚ 3.62890625 disk.sdb.in‚ 3.49609375 disk.sdb2.in‚ 3.49609375 proc.cpu.threads‚17 network.sent‚ 1.3509617e+07# disk.nvme0n1.out‚35773.33984375 disk.sdb2.out‚ 695.87109375$ memory_percent‚36.08567714691162) disk./.usagePercent‚83.14452090392838 disk.sdb.out‚ 695.87109375 network.recv‚ 2.138597e+06 disk.sda.in‚ 3.62890625 disk.sda1.out‚0 disk.sda2.out‚ 695.87109375 disk.rbd2.in‚0# proc.memory.rssMB‚ 3030.74609375 cpu‚25.17082088840151 disk.sdb1.in‚0 disk.dm-0.out‚81455.27734375 disk.md0.in‚7.125 disk.rbd2.out‚ 1.62109375) proc.memory.percent‚18.49820613861084 disk.sda.out‚ 695.87109375 disk.nvme0n1.in‚ 328983.21875 disk.md0p1.in‚7.125 disk.rbd0.in‚0.015625ÂÏÛ‡‚(:… ÀÈçÔ†£­‡ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚99.541' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚24.88525 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v{coˆ‚(:ä ÀÈçԆ䯈 disk.rbd2.out‚ 1.62109375 disk.sdb.out‚ 697.830078125 disk.sdb1.in‚0 disk.md0p1.out‚ 672.87890625$ memory_percent‚36.08407974243164 disk.md0.out‚ 672.87890625 disk.sda.out‚ 697.830078125 disk.sda1.in‚0 disk.sda2.in‚ 3.62890625" disk.nvme1n1.in‚200730.4765625# disk.nvme1n1.out‚45768.04296875) disk./.usagePercent‚83.15102527649823% disk./.usageGB‚2973.8834266662598 disk.sdb2.in‚ 3.49609375 disk.md0p1.in‚7.125( proc.memory.availableMB‚ 10471.984375 disk.rbd0.in‚0.015625 disk.sdb.in‚ 3.49609375 disk.sda.in‚ 3.62890625 disk.dm-0.out‚ 81599.03125) proc.memory.percent‚18.49820613861084 cpu‚25.17038246173034 disk.nvme0n1.in‚ 328983.21875 proc.cpu.threads‚17 disk.sdb2.out‚ 697.830078125 disk.rbd0.out‚ 1.68359375# disk.nvme0n1.out‚35830.98828125# proc.memory.rssMB‚ 3030.74609375 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.sda2.out‚ 697.830078125 network.sent‚ 1.3515756e+07 network.recv‚ 2.142782e+06 disk.md0.in‚7.125 disk.dm-0.in‚529713.6953125 disk.sda1.out‚0$Œ«2‰- _wandbruntime‚6404  _runtime‚6404¥¹ÆŠÀ trainer/global_step‚5499 $ _timestamp‚1.788470351753329e+09 _runtime‚6404.938394955 _step‚448 & lr-AdamW/pg1‚0.0005466535450042366 & lr-AdamW/pg2‚0.0005466535450042366øgÄË‹Å _runtime‚6404.938394955 _step‚448 & lr-AdamW/pg1‚0.0005466535450042366 & lr-AdamW/pg2‚0.0005466535450042366 trainer/global_step‚5499 $ _timestamp‚1.788470351753329e+09À¸¬t2Œ- _wandbruntime‚6404  _runtime‚6404àˆXí{çO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚223667505173504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚248039000044032 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚212871586481104 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10891407648986 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚ 6404.93949195 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10922136064512 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62363584 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 102001211 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚248038698067456 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 149415100 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39636924 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚223667505173504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 trainer/global_step‚5499 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚234589874906624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11108879471616 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚223667487170560 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 253842699 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 104401253 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚703 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301976576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 278036480 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 104401253 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚234589874906624 TGDeviceStatsMonitor.on_train_batch_st§OVprart/inactive_split.large_pool.freed‚62363582 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104401255 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10922136064512 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39637627 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚223762765149006 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚223763011169450 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚212871603520464 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 102000506 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚234589623235072 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 149441444 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 253816353 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10922369733120 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104401255 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10922136064512 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39637627 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚223762765149006 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚223763011169450 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚212871603520464 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 102000506 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚234589623235072 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 149441444 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 253816353 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10922369733120 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 149415100 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39636924 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚223667505173504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 trainer/global_step‚5499 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚234589874906624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11108879471616 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚223667487170560 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 253842699 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 104401253 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚703 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301976576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 278036480 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 104401253 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309ÁVd5‚(:… ÏÈçÔ£ò† gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚120.31' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.0775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-˜‹j‚(:ß ÏÈçÔ¦±·ˆ network.recv‚ 2.146552e+06 disk.sda1.in‚0 disk.md0p1.out‚ 674.21484375 disk.sdb1.out‚0# proc.memory.rssMB‚ 3030.74609375 proc.cpu.threads‚17) disk./.usagePercent‚83.15641163337037 disk.dm-0.in‚529713.6953125 disk.rbd0.in‚0.015625 disk.sdb.in‚ 3.49609375$ disk./.usageGB‚2974.076068878174 disk.sda.in‚ 3.62890625 disk.sda.out‚ 699.2109375 disk.sda1.out‚0 disk.sda2.out‚ 699.2109375 disk.nvme0n1.in‚ 328983.21875 disk.sdb2.in‚ 3.49609375 disk.sdb2.out‚ 699.2109375 disk.rbd0.out‚ 1.68359375 disk.sdb1.in‚0# disk.nvme0n1.out‚35859.13671875 disk.dm-0.out‚ 81713.2421875 disk.md0.in‚7.125 disk.md0.out‚ 674.21484375 disk.rbd2.in‚0 disk.rbd2.out‚ 1.62109375# disk.nvme1n1.out‚45854.10546875 disk.sda2.in‚ 3.62890625 disk.sdb.out‚ 699.2109375) proc.memory.availableMB‚ 10471.9765625 disk.md0p1.in‚7.125" disk.nvme1n1.in‚200730.4765625$ memory_percent‚36.08412742614746) proc.memory.percent‚18.49820613861084 cpu‚25.169946187496866 network.sent‚ 1.3531587e+07—6?€2‘- _wandbruntime‚6405  _runtime‚6405ké¥*÷&’ñM K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104419528 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚234631124982784 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚212909116838528 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚248082378607616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10924333772288 $ _timestamp‚1.788470352733655e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 149468162 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10924100090368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 253887692 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102018602 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39644218 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚450 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 253861320 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚223706791210496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11110598005248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚212909133877888 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚223706791210496 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚236971804542464 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10893137156888 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚234630873297920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 149441792 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚236971780602368 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 149441792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 253861320 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚223802253995416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 102019293 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 104419530 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62374384 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 253887692 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10924100090368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚689 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11110884417024 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚223802500016068 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚234631124982784 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚212909116838528 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚248082378607616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10924333772288 $ _timestamp‚1.788470352733655e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 149468162 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10924100090368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 253887692 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102018602 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39644218 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚450 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 253861320 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚223706791210496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11110598005248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚212909133877888 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚223706791210496 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚236971804542464 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10893137156888 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚234630873297920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 149—²´ 441792 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚236971780602368 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 149441792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 253861320 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚223802253995416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 102019293 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 104419530 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62374384 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 253887692 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10924100090368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚689 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11110884417024 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ¯”_2”- _wandbruntime‚6405  _runtime‚6405…_•Ù 1loss/SRqqt3/detection_loss‚0.8602614039248162 /loss/FRt3/assignment_loss‚1.382723262310028 2loss/SRqqt3/assignment_loss‚0.9070136626865101 1loss/SRqqt4/detection_loss‚2.1838821119732326 % _timestamp‚1.7884703527358022e+09 0loss/FBt2/assignment_loss‚0.5377235728869475 /loss/FRt2/detection_loss‚2.3591290608455893 /loss/FRt3/detection_loss‚1.2964982738097508 1loss/SRqqt2/detection_loss‚2.2282677786914924 /loss/FBt4/detection_loss‚2.7666614651679993 $loss/total_loss‚41.7048437215475 2loss/SRqqt1/assignment_loss‚1.3201143030880713 2loss/SRqqt4/assignment_loss‚1.7389604681067996 1loss/FBt4/assignment_loss‚0.47321670670923294 1loss/SRqqt1/detection_loss‚1.8620343582302916 .loss/FBt2/detection_loss‚2.290043751398722 epoch‚4 1loss/FBt1/assignment_loss‚0.08384579968676557 2loss/FBt3/assignment_loss‚0.025703252872125024 .loss/FRt1/detection_loss‚2.842531901597977 /loss/FRt4/detection_loss‚2.8633338789145153 _runtime‚6405.920713256 0loss/FRt4/assignment_loss‚3.1909948587417603 2loss/SRqqt2/assignment_loss‚1.4641099067306833 trainer/global_step‚5499 /loss/FRt1/assignment_loss‚3.012387559811274 0loss/FRt2/assignment_loss‚3.1939834997057917 /loss/FBt1/detection_loss‚2.3703200817108154 0loss/FBt3/detection_loss‚0.45110280194630226 _step‚451ªÜ*äd–Þ _runtime‚6405.920713256 0loss/FRt4/assignment_loss‚3.1909948587417603 2loss/SRqqt2/assignment_loss‚1.4641099067306833 trainer/global_step‚5499 /loss/FRt1/assignment_loss‚3.012387559811274 0loss/FRt2/assignment_loss‚3.1939834997057917 /loss/FBt1/detection_loss‚2.3703200817108154 0loss/FBt3/detection_loss‚0.45110280194630226 _step‚451 1loss/SRqqt3/detection_loss‚0.8602614039248162 /loss/FRt3/assignment_loss‚1.382723262310028 2loss/SRqqt3/assignment_loss‚0.9070136626865101 1loss/SRqqt4/detection_loss‚2.1838821119732326 % _timestamp‚1.7884703527358022e+09 0loss/FBt2/assignment_loss‚0.5377235728869475 /loss/FRt2/detection_loss‚2.3591290608455893 /loss/FRt3/detection_loss‚1.2964982738097508 1loss/SRqqt2/detection_loss‚2.2282677786914924 /loss/FBt4/detection_loss‚2.7666614651679993 $loss/total_loss‚41.7048437215475 2loss/SRqqt1/assignment_loss‚1.3201143030880713 2loss/SRqqt4/assignment_loss‚1.7389604681067996 1loss/FBt4/assignment_loss‚0.47321670670923294 1loss/SRqqt1/detection_loss‚1.8620343582302916 .loss/FBt2/detection_loss‚2.290043751398722 epoch‚4 1loss/FBt1/assignment_loss‚0.08384579968676557 2loss/FBt3/assignment_loss‚0.025703252872125024 .loss/FRt1/detection_loss‚2.842531901597977 /loss/FRt4/detection_loss‚2.8633338789145153ÃV„h5’—‚(:‡ ÞÈçÔÀó¾‡ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚119.091' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.77275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×´=jz˜‚(:ï ÞÈçÔüâÖˆ disk.dm-0.in‚529713.6953125 cpu‚25.169510398806757 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.sda.in‚ 3.62890625# disk.nvme0n1.out‚35986.70703125 disk.md0.out‚ 675.80859375 disk.rbd2.in‚0 disk.rbd2.out‚ 1.62109375% disk./.usageGB‚2974.0740814208984! disk.sda2.out‚700.86181640625# proc.memory.rssMB‚ 3030.74609375 disk.sdb.in‚ 3.49609375 disk.sda2.in‚ 3.62890625 disk.md0p1.in‚7.125 disk.rbd0.in‚0.015625) proc.memory.availableMB‚ 10471.3046875) proc.memory.percent‚18.49820613861084" disk.nvme1n1.in‚200730.4765625 disk.sdb1.in‚0 disk.dm-0.out‚81847.03515625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb2.in‚ 3.49609375 disk.rbd0.out‚ 1.68359375 network.recv‚ 2.153112e+06) disk./.usagePercent‚83.15635606323313 disk.md0p1.out‚ 675.80859375 disk.md0.in‚7.125$ memory_percent‚36.08822822570801 disk.sdb.out‚700.86181640625 disk.sda.out‚700.86181640625 disk.nvme0n1.in‚ 328983.21875 network.sent‚ 1.3628169e+07! disk.nvme1n1.out‚ 45860.328125! disk.sdb2.out‚700.86181640625¼@ÀO’™‚(:‡ íÈçÔú²©‡ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚117.193' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.29825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0UËeš‚(:Ú íÈçÔ£Ùˆ" disk.nvme1n1.out‚ 45926.8828125 disk.rbd0.out‚ 1.68359375) disk./.usagePercent‚83.15985100888918 disk.sda1.out‚0 disk.md0p1.in‚7.125* proc.memory.availableMB‚10472.02734375 cpu‚25.1691227002026" disk.nvme1n1.in‚200730.4765625 disk.sda2.in‚ 3.62890625 disk.sda2.out‚ 702.734375 disk.sda.out‚ 702.734375# disk.nvme0n1.out‚36124.40234375 disk.dm-0.in‚529713.6953125 network.sent‚ 1.3712232e+07 network.recv‚ 2.159332e+06 disk.sda.in‚ 3.62890625 disk.md0p1.out‚ 677.61328125 disk.sdb2.in‚ 3.49609375 disk.sdb2.out‚ 702.734375$ disk./.usageGB‚2974.199077606201 disk.sdb.out‚ 702.734375 disk.sdb1.out‚0 disk.nvme0n1.in‚ 328983.21875 disk.dm-0.out‚82051.28515625 disk.sda1.in‚0) proc.memory.percent‚18.49820613861084 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.md0.in‚7.125 disk.rbd0.in‚0.015625$ memory_percent‚36.08381748199463 disk.rbd2.in‚0 disk.rbd2.out‚ 1.62109375 disk.sdb.in‚ 3.49609375 disk.md0.out‚ 677.61328125# proc.memory.rssMB‚ 3030.74609375d_7&œ›‚(:‘ üÈçÔŒ¬ž† gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™¥·Ìhœ‚(:Ý üÈçÔÜšî† disk.sda2.out‚ 704.3515625" disk.nvme1n1.in‚200730.4765625! disk.nvme1n1.out‚ 45993.640625 disk.sdb.in‚ 3.49609375 network.recv‚ 2.163598e+06# disk.nvme0n1.out‚36167.52734375 disk.sda1.out‚0 disk.sdb.out‚ 704.3515625) proc.memory.percent‚18.49820613861084) disk./.usagePercent‚83.15565605016083! disk.nvme0n1.in‚ 328983.265625 disk.sdb2.in‚ 3.49609375% memory_percent‚36.086082458496094 network.sent‚ 1.3718244e+07$ disk./.usageGB‚2974.049045562744 disk.dm-0.in‚529713.7421875 disk.sda1.in‚0 disk.sdb2.out‚ 704.3515625 disk.rbd2.out‚ 1.62109375 disk.sda.out‚ 704.3515625 disk.dm-0.out‚82161.16796875 disk.md0p1.out‚ 679.17578125 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 1.68359375 disk.sda.in‚ 3.62890625# proc.memory.rssMB‚ 3030.74609375 cpu‚25.16866367590035 proc.cpu.threads‚17 disk.md0p1.in‚7.125 disk.sdb1.in‚0' proc.memory.availableMB‚ 10471.65625 disk.sda2.in‚ 3.62890625 disk.md0.in‚7.125 disk.sdb1.out‚0 disk.md0.out‚ 679.17578125 disk.rbd2.in‚0‚ß·2- _wandbruntime‚6456  _runtime‚6456&éµYÅž¿ % lr-AdamW/pg1‚0.000541014084432702 % lr-AdamW/pg2‚0.000541014084432702 trainer/global_step‚5549 % _timestamp‚1.7884704028515441e+09 _runtime‚6456.036487316 _step‚452ÚŽLÊŸÄ % lr-AdamW/pg1‚0.000541014084432702 % lr-AdamW/pg2‚0.000541014084432702 trainer/global_step‚5549 % _timestamp‚1.7884704028515441e+09 _runtime‚6456.036487316 _step‚452ÄR"»2 - _wandbruntime‚6456  _runtime‚64561!¬~î'¡èO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚236652140791296 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 102887777 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150779396 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚225631807023104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11020351771136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150779396 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚225631789020160 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 _runtime‚6456.037430631 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚236652140791296 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105315005 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62903684 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39984093 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚239014319074816 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 256094401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39983418 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚677 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚225631807023104 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚250222884449280 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11208875739648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150753052 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚5549 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11020351771136 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 102887100 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚250223194814464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11020585439744 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273œ‡Q½ó'¢íO O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150779396 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚225631807023104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11020351771136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150779396 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚225631789020160 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 _runtime‚6456.037430631 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚236652140791296 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105315005 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62903684 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39984093 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚239014319074816 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 256094401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39983418 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚677 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚225631807023104 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚250222884449280 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11208875739648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150753052 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚5549 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11020351771136 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 102887100 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚250223194814464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11020585439744 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚236652140791296 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 102887777 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1Å&c»ñ2£- _wandbruntime‚6456  _runtime‚6456ç¨N̤òM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102904492 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62914484 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11210870007296 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_en:èf¬,"d/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150779744 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11022315796992 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10991075197396 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚225671075057152 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11022315796992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚671 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62914486 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚225671093060096 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11022549478912 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚239056003044864 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚236693390854144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 256139394 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105333280 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚5549 _step‚454 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10991075197396 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚225671075057152 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11022315796992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚671 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62914486 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚225671093060096 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11022549478912 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚239056003044864 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚236693390854144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 256139394 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105333280 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚5549 _step‚454 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102904492 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62914484 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11210870007296 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150779744 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11022315796992 PPÞª‚(:… ‹ÉçÔ§ö²‡ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚141.704' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.426 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ŒsÓ’«‚(:‡ šÉçÔÍ»©† gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚137.083' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.27075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z9Þ°r¬‚(:ç šÉçÔç†è†" disk.nvme1n1.in‚200730.4765625 disk.sdb1.in‚0 disk.dm-0.in‚529713.7421875! disk.sda2.out‚707.73583984375$ memory_percent‚36.08725070953369 disk.sda.in‚ 3.62890625 network.recv‚ 2.174169e+06! disk.sdb2.out‚707.73583984375 disk.sda1.in‚0) proc.memory.percent‚18.49820613861084) disk./.usagePercent‚83.16509966701435% disk./.usageGB‚2974.3867950439453 disk.md0p1.in‚7.125 disk.md0p1.out‚682.4375 disk.rbd0.out‚ 1.68359375# disk.nvme1n1.out‚46195.66796875 disk.md0.in‚7.125 disk.sdb.out‚707.73583984375 disk.sdb1.out‚0 disk.sda.out‚707.73583984375* proc.memory.availableMB‚10471.46484375 disk.md0.out‚682.4375 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.rbd2.out‚ 1.62109375 disk.sda2.in‚ 3.62890625# proc.memory.rssMB‚ 3030.74609375 cpu‚25.167772113332674 disk.sdb2.in‚ 3.49609375 disk.sdb.in‚ 3.49609375! disk.nvme0n1.in‚ 328983.265625 proc.cpu.threads‚17# disk.nvme0n1.out‚36240.64453125 disk.sda1.out‚0 disk.dm-0.out‚ 82436.3125 network.sent‚ 1.3831258e+07¼åv‘­‚(:† ©ÉçÔª¤”† gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚138.642' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.6605 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09‰dx®‚(:í ©ÉçÔÅ­ž‡) proc.memory.percent‚18.49820613861084 network.sent‚ 1.3837264e+07% disk./.usageGB‚2974.5740280151367 disk.sda1.out‚0' proc.memory.availableMB‚ 10471.71875# disk.nvme1n1.out‚46262.26171875 disk.sda.in‚ 3.62890625 disk.md0.in‚7.125 disk.rbd0.in‚0.015625" disk.nvme1n1.in‚200730.4765625 disk.sdb.in‚ 3.49609375 disk.sdb1.in‚0 disk.sda1.in‚0 disk.sda2.in‚ 3.62890625! disk.sda2.out‚709.29736328125 disk.sdb1.out‚0 disk.dm-0.in‚529713.7421875 proc.cpu.threads‚17 disk.sdb2.in‚ 3.49609375! disk.sdb2.out‚709.29736328125 disk.rbd0.out‚ 1.68359375 disk.dm-0.out‚82585.78515625 network.recv‚ 2.17863e+06) disk./.usagePercent‚83.17033477925196 disk.sdb.out‚709.29736328125 disk.sda.out‚709.29736328125! disk.nvme0n1.in‚ 328983.265625$ memory_percent‚36.08570098876953 cpu‚25.16730655392245" disk.nvme0n1.out‚ 36323.5234375 disk.md0p1.in‚7.125 disk.md0.out‚ 683.94140625 disk.rbd2.in‚0 disk.rbd2.out‚ 1.62109375 disk.md0p1.out‚ 683.94140625# proc.memory.rssMB‚ 3030.74609375 Åæ2¯- _wandbruntime‚6506  _runtime‚6506^ï Ç°Á & lr-AdamW/pg1‚0.0005353477846905664 & lr-AdamW/pg2‚0.0005353477846905664 trainer/global_step‚5599 % _timestamp‚1.7884704533142092e+09 _runtime‚6506.499229792 _step‚4566¦Ì±Æ & lr-AdamW/pg2‚0.0005353477846905664 trainer/global_step‚5599 % _timestamp‚1.7884704533142092e+09 _runtime‚6506.499229792 _step‚456 & lr-AdamW/pg1‚0.0005353477846905664Èßíâ2²- _wandbruntime‚6506  _runtime‚6506 €ˆÕn³èO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 152091004 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 106228755 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚238714658347520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚5599 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚227596090869760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282230784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 152117348 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40328895 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306170880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 258319757 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚216624639262864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 106228753 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚457 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11308733935104 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚227711922992866 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚238714658347520 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11118801146368 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚241098517577216 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚227711676972422 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚227596108872704 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚252407251512320 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚238714910019072 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚241098493637120 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚216624622223504 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚238714910019072 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 106228755 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11087283730002 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11308451704320 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 106228753 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚227596090869760 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚216624622223504 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚238714910019072 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 106228755 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11087283730002 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11308451704320 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 106228753 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚227596090869760 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 106228755 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚238714658347520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚5599 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚227596090869760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282230784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 152117348 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40328895 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306170880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 258319757 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚216624639262864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 106228753 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚457 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11308733935104 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚227711922992866 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚238714658347520 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11118801146368 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚241098517577216 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚227711676972422 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚227596108872704 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚252407251512320 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚238714910019072 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚241098493637120 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103790143 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 258364724 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11089013237904 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚227751165818832 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 152117696 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40336247 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106247028 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 258391096 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚252450966204928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚227635394909696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63454586 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚238755908410368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚688 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚241140201547264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚227635394909696 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚227635376906752 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 106247030 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚252450647464448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚241140177607168 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚6507.543787078 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11089242219196 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 258391096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 trainer/global_step‚5599 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 258364724 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚216662152580928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106247028 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 152144066 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 % _timestamp‚1.7884704543578978e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 152144066 _step‚458 E7L@ ·÷M H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 258364724 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚216662152580928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106247028 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 152144066 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 % _timestamp‚1.7884704543578978e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 152144066 _step‚458 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103790143 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 258364724 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11089013237904 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚227751165818832 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 152117696 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40336247 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106247028 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 258391096 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚252450966204928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚227635394909696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63454586 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚238755908410368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚688 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚241140201547264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚227635394909696 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚227635376906752 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 106247030 XDDeviceStatsMonitor.on_train_batch_end/inactive_spt„‚`½lit_bytes.all.freed‚252450647464448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚241140177607168 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚6507.543787078 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11089242219196 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 258391096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 trainer/global_step‚5599ÊvÓ¤v2¸- _wandbruntime‚6507  _runtime‚6507~¨^}X¹Ò /loss/FBt2/detection_loss‚2.9930676562445506 2loss/SRqqt1/assignment_loss‚1.5854555054547939 2loss/SRqqt3/assignment_loss‚1.0902265425187996 1loss/FBt4/assignment_loss‚0.31353307212239656 %loss/total_loss‚43.98566430538267 /loss/SRqqt2/detection_loss‚2.00688289578368 /loss/FBt3/detection_loss‚0.7014626641757786 _runtime‚6507.545059673 0loss/FRt1/assignment_loss‚3.3274434709548952 .loss/FRt1/detection_loss‚3.141088242530823 /loss/FRt4/assignment_loss‚3.364341612519889 1loss/SRqqt2/assignment_loss‚1.662619562047284 0loss/FBt2/assignment_loss‚0.3578523867763579 .loss/FBt4/detection_loss‚2.036439824849367 /loss/FRt2/assignment_loss‚3.552366694918385 2loss/SRqqt4/assignment_loss‚1.6258909019331138 /loss/FRt3/detection_loss‚1.6125429205130786 /loss/FBt1/detection_loss‚2.1340551376342773 /loss/FRt4/detection_loss‚2.9416886732496064 1loss/SRqqt4/detection_loss‚2.4362865313887596 epoch‚4 _step‚459 /loss/FRt3/assignment_loss‚1.498587816953659 0loss/FBt1/assignment_loss‚0.2366410343813507 1loss/SRqqt3/detection_loss‚1.1728216496980692 trainer/global_step‚5599 % _timestamp‚1.7884704543602314e+09 1loss/FBt3/assignment_loss‚0.01986448559843834 /loss/FRt2/detection_loss‚2.1548511131172914 0loss/SRqqt1/detection_loss‚2.019653910018028ºh2‡]º× /loss/FRt4/detection_loss‚2.9416886732496064 1loss/SRqqt4/detection_loss‚2.4362865313887596 epoch‚4 _step‚459 /loss/FRt3/assignment_loss‚1.498587816953659 0loss/FBt1/assignment_loss‚0.2366410343813507 1loss/SRqqt3/detection_loss‚1.1728216496980692 trainer/global_step‚5599 % _timestamp‚1.7884704543602314e+09 1loss/FBt3/assignment_loss‚0.01986448559843834 /loss/FRt2/detection_loss‚2.1548511131172914 0loss/SRqqt1/detection_loss‚2.019653910018028 /loss/FBt2/detection_loss‚2.9930676562445506 2loss/SRqqt1/assignment_loss‚1.5854555054547939 2loss/SRqqt3/assignment_loss‚1.0902265425187996 1loss/FBt4/assignment_loss‚0.31353307212239656 %loss/total_loss‚43.98566430538267 /loss/SRqqt2/detection_loss‚2.00688289578368 /loss/FBt3/detection_loss‚0.7014626641757786 _runtime‚6507.545059673 0loss/FRt1/assignment_loss‚3.3274434709548952 .loss/FRt1/detection_loss‚3.141088242530823 /loss/FRt4/assignment_loss‚3.364341612519889 1loss/SRqqt2/assignment_loss‚1.662619562047284 0loss/FBt2/assignment_loss‚0.3578523867763579 .loss/FBt4/detection_loss‚2.036439824849367 /loss/FRt2/assignment_loss‚3.552366694918385 2loss/SRqqt4/assignment_loss‚1.6258909019331138 /loss/FRt3/detection_loss‚1.6125429205130786 /loss/FBt1/detection_loss‚2.1340551376342773Ë@yÁ»‚(:… ¸ÉçÔÏôö‡ gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚116.916' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.229 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹PÆv¼‚(:ë ¸ÉçÔâß«ˆ# proc.memory.rssMB‚ 3030.74609375) proc.memory.percent‚18.49820613861084 disk.sdb.in‚ 3.49609375 disk.sdb1.in‚0 network.sent‚ 1.3855983e+07 network.recv‚ 2.182094e+06 disk.sda1.in‚0! disk.sda2.out‚710.84423828125 disk.sdb2.in‚ 3.49609375! disk.sdb2.out‚710.84423828125 disk.rbd0.out‚ 1.68359375 disk.rbd2.in‚0 disk.rbd2.out‚ 1.62109375* proc.memory.availableMB‚10471.30859375 cpu‚25.166914125017335) disk./.usagePercent‚83.18431498851821 disk.sda2.in‚ 3.62890625 disk.sdb.out‚710.84423828125 disk.dm-0.in‚529713.7421875 disk.md0.in‚7.125 disk.sda.out‚710.84423828125 proc.cpu.threads‚17 disk.nvme1n1.out‚ 46390.625 disk.sdb1.out‚0% disk./.usageGB‚2975.0740280151367 disk.dm-0.out‚ 82834.9921875 disk.md0p1.in‚7.125 disk.md0p1.out‚ 685.43359375" disk.nvme1n1.in‚200730.4765625 disk.sda.in‚ 3.62890625" disk.nvme0n1.out‚ 36444.3671875 disk.md0.out‚ 685.43359375! disk.nvme0n1.in‚ 328983.265625 disk.sda1.out‚0 disk.rbd0.in‚0.015625# memory_percent‚36.0882043838501ŽŽ©’½‚(:‡ ÇÉçÔ«‚ß… gpu.0.gpu‚10.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚124.321' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.08025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®Kw¾‚(:ì ÇÉçÔõ¨Ú† disk.rbd0.in‚0.015625$ memory_percent‚36.08689308166504 cpu‚25.166545521694378) disk./.usagePercent‚83.18431658842619 disk.sdb1.out‚0 disk.sda1.out‚0! disk.sdb2.out‚712.21923828125 disk.sdb.in‚ 3.49609375! disk.nvme0n1.in‚ 328983.265625) proc.memory.availableMB‚ 10471.5234375# proc.memory.rssMB‚ 3030.74609375 network.sent‚ 1.3949092e+07 disk.sda.in‚ 3.62890625! disk.nvme0n1.out‚ 36622.296875 disk.sda2.in‚ 3.62890625! disk.sda2.out‚712.21923828125 disk.md0p1.in‚7.125 disk.rbd0.out‚ 1.68359375 disk.dm-0.out‚83084.45703125 disk.sda1.in‚0 disk.md0.in‚7.125 disk.rbd2.out‚ 1.62109375" disk.nvme1n1.in‚200730.4765625% disk./.usageGB‚2975.0740852355957# disk.nvme1n1.out‚46462.16015625 proc.cpu.threads‚17 network.recv‚ 2.188168e+06 disk.sdb.out‚712.21923828125 disk.sdb1.in‚0 disk.sda.out‚712.21923828125 disk.dm-0.in‚529713.7421875 disk.md0.out‚ 686.765625) proc.memory.percent‚18.49820613861084 disk.rbd2.in‚0 disk.md0p1.out‚ 686.765625 disk.sdb2.in‚ 3.49609375DŸ©Á¿‚(:… ÖÉçÔÅ⇠gpu.0.gpu‚7.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚145.438' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.3595 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ø|¼PtÀ‚(:é ÖÉçÔÍüéˆ disk.sdb.in‚ 3.49609375# disk.nvme0n1.out‚36665.26953125 disk.sda1.in‚0% memory_percent‚36.086153984069824 disk.rbd2.out‚ 1.62109375! disk.nvme1n1.out‚ 46595.265625 disk.dm-0.in‚529713.7421875 disk.dm-0.out‚83260.53515625 disk.sda.in‚ 3.62890625 disk.sda1.out‚0* proc.memory.availableMB‚10471.64453125 cpu‚25.166186513964988 network.sent‚ 1.395548e+07) disk./.usagePercent‚83.17662657738127% disk./.usageGB‚2974.7990531921387 disk.sdb1.out‚0 disk.sda.out‚ 714.259765625 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.sdb1.in‚0) proc.memory.percent‚18.49820613861084 disk.sda2.in‚ 3.62890625 disk.sdb2.in‚ 3.49609375 disk.sdb2.out‚ 714.259765625 disk.rbd0.out‚ 1.68359375 disk.sdb.out‚ 714.259765625! disk.nvme0n1.in‚ 328983.265625 disk.sda2.out‚ 714.259765625 disk.md0p1.in‚7.125 disk.md0p1.out‚ 688.75390625 disk.md0.in‚7.125 disk.md0.out‚ 688.75390625" disk.nvme1n1.in‚200730.4765625# proc.memory.rssMB‚ 3030.74609375 proc.cpu.threads‚17 network.recv‚ 2.192551e+06´x¿r’Á‚(:‡ åÉçÔ†çᇠgpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚139.251' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.81275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þ·@x‚(:í åÉçÔ¯ÊÔˆ$ disk./.usageGB‚2974.822368621826 disk.sdb.out‚715.84033203125 disk.md0.out‚ 690.28125 disk.rbd2.out‚ 1.62109375 network.sent‚ 1.3963542e+07( disk./.usagePercent‚83.1772784865536 disk.sdb.in‚ 3.49609375 disk.sda.in‚ 3.62890625 disk.sda2.in‚ 3.62890625 proc.cpu.threads‚17" disk.nvme1n1.in‚200730.4765625 disk.dm-0.in‚529714.73046875 disk.md0p1.in‚7.125! disk.sdb2.out‚715.84033203125 disk.rbd0.in‚0.015625 cpu‚25.165809516515605 network.recv‚ 2.196992e+06 disk.sdb1.out‚0 disk.sda1.in‚0% memory_percent‚36.086153984069824# disk.nvme0n1.in‚328984.25390625! disk.sda2.out‚715.84033203125) proc.memory.percent‚18.49820613861084 disk.sda.out‚715.84033203125 disk.sdb2.in‚ 3.49609375 disk.rbd2.in‚0# proc.memory.rssMB‚ 3030.74609375# disk.nvme1n1.out‚46794.85546875 disk.dm-0.out‚ 83460.671875 disk.md0p1.out‚ 690.28125 disk.md0.in‚7.125* proc.memory.availableMB‚10471.64453125# disk.nvme0n1.out‚36665.81640625 disk.sdb1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚ 1.68359375A@?2Ã- _wandbruntime‚6560  _runtime‚6560÷®Ÿ ÇÄÁ _runtime‚6560.410816556 _step‚460 & lr-AdamW/pg1‚0.0005296558266262035 & lr-AdamW/pg2‚0.0005296558266262035 trainer/global_step‚5649 % _timestamp‚1.7884705072257555e+09ª×o8ÌÅÆ trainer/global_step‚5649 % _timestamp‚1.7884705072257555e+09 _runtime‚6560.410816556 _step‚460 & lr-AdamW/pg1‚0.0005296558266262035 & lr-AdamW/pg2‚0.0005296558266262035ÌØëà2Æ- _wandbruntime‚6560  _runtime‚6560Xé( î'ÇèO @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚240777175903744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40676162 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 153428956 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 153455300 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63983884 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚240777427575296 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚229560410722304 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 104660731 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚240777175903744 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚229686132884130 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11185221770510 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 153428956 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚229560392719360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11216783184384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40676847 _step‚461 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11217016852992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 260597805 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚254591466070016 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚229560392719360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚243182692139520 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11408463565312 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 153455300 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚243182716079616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104660044 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚229686378904574 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 153428956 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 153455300 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63983884 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚240777427575296 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚229560410722304 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 104660731 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚240777175903744 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚229686132884130 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11185221770510 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 153428956 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚229560392719360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11216783184384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40676847 _step‚461 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11217016852992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 260597805 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚254591466070016 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚229560392719360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚243182692139520 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11408463565312 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 153455300 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚243182716079616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104660044 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚229686378904574 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚229599696759296 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260642798 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚229599678756352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 153482018 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11186951278412 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _step‚462 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107160778 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40683477 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 153455648 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11218747210240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚229725867751192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚684 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5649 _runtime‚6561.431606395 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104677479 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63994686 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107160778 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107160780 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚243224400049664 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚229599678756352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚218538687491488 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 260616426 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚240818425966592 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚240818677651456 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11187180259704 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11410744404480ð>ü½ý&Ë÷M WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚229599678756352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚218538687491488 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 260616426 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚240818425966592 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚240818677651456 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11187180259704 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11410744404480 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11218980892160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 260616426 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63994684 % _timestamp‚1.7884705082450278e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚240818677651456 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11218980892160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 260642798 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚229599696759296 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260642798 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚229599678756352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 153482018 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11186951278412 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _step‚462 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107160778 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40683477 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 153455648 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11218747210240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚229725867751192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚684 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5649 _runtime‚6561.431606395 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104677479 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63994686 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107160778 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107160780 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚243224400049664 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096Ψ歖2Ì- _wandbruntime‚6561  _runtime‚6561íº+ÌbÍÜ 1loss/SRqqt2/assignment_loss‚1.568028309055277 /loss/FRt3/detection_loss‚1.3855071116606115 %loss/total_loss‚41.21942923572869 0loss/FRt4/assignment_loss‚2.2977182927884554 1loss/FBt1/assignment_loss‚0.03610221560902005 3loss/FBt2/assignment_loss‚0.0017825385062031694 1loss/SRqqt3/detection_loss‚0.9139558691755835 1loss/SRqqt4/detection_loss‚2.2490141605719542 0loss/FRt3/assignment_loss‚1.3124920272491347 1loss/SRqqt1/detection_loss‚1.9792287009102958 /loss/FBt4/detection_loss‚2.5448664501309395 /loss/FBt2/detection_loss‚2.3812844111369205 _step‚463 0loss/FRt1/assignment_loss‚3.1252912396476384 2loss/SRqqt4/assignment_loss‚1.2774780960037158 /loss/FRt1/detection_loss‚2.5233134045487358 /loss/FRt4/detection_loss‚3.4624291940739282 1loss/SRqqt2/detection_loss‚2.3776694955052555 _runtime‚6561.433963385 2loss/SRqqt3/assignment_loss‚1.0950788624405143 trainer/global_step‚5649 # _timestamp‚1.78847050824907e+09 2loss/SRqqt1/assignment_loss‚1.6251663715407556 1loss/FBt3/assignment_loss‚0.03029768563788486 /loss/FRt2/detection_loss‚2.3066275844537856 /loss/FBt3/detection_loss‚0.5761524029894844 epoch‚4 0loss/FRt2/assignment_loss‚3.6101480082767767 1loss/FBt4/assignment_loss‚0.10321590743825482 /loss/FBt1/detection_loss‚2.4365808963775635ClôgÎá 0loss/FRt1/assignment_loss‚3.1252912396476384 2loss/SRqqt4/assignment_loss‚1.2774780960037158 /loss/FRt1/detection_loss‚2.5233134045487358 /loss/FRt4/detection_loss‚3.4624291940739282 1loss/SRqqt2/detection_loss‚2.3776694955052555 _runtime‚6561.433963385 2loss/SRqqt3/assignment_loss‚1.0950788624405143 trainer/global_step‚5649 # _timestamp‚1.78847050824907e+09 2loss/SRqqt1/assignment_loss‚1.6251663715407556 1loss/FBt3/assignment_loss‚0.03029768563788486 /loss/FRt2/detection_loss‚2.3066275844537856 /loss/FBt3/detection_loss‚0.5761524029894844 epoch‚4 0loss/FRt2/assignment_loss‚3.6101480082767767 1loss/FBt4/assignment_loss‚0.10321590743825482 /loss/FBt1/detection_loss‚2.4365808963775635 1loss/SRqqt2/assignment_loss‚1.568028309055277 /loss/FRt3/detection_loss‚1.3855071116606115 %loss/total_loss‚41.21942923572869 0loss/FRt4/assignment_loss‚2.2977182927884554 1loss/FBt1/assignment_loss‚0.03610221560902005 3loss/FBt2/assignment_loss‚0.0017825385062031694 1loss/SRqqt3/detection_loss‚0.9139558691755835 1loss/SRqqt4/detection_loss‚2.2490141605719542 0loss/FRt3/assignment_loss‚1.3124920272491347 1loss/SRqqt1/detection_loss‚1.9792287009102958 /loss/FBt4/detection_loss‚2.5448664501309395 /loss/FBt2/detection_loss‚2.3812844111369205 _step‚463ÏÊØ^Ÿ‘Ï‚(:† ôÉçÔ›±† gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚136.134' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.0335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ý"ÝgЂ(:Ü ôÉçÔ÷ûㆠdisk.dm-0.out‚ 83662.6875 disk.sdb.in‚ 3.49609375) disk./.usagePercent‚83.17820397999083 disk./.usageGB‚ 2974.85546875 disk.md0p1.in‚7.125 disk.md0p1.out‚ 691.7265625 disk.md0.in‚7.125 disk.rbd0.out‚ 1.68359375# proc.memory.rssMB‚ 3030.74609375 network.recv‚ 2.201475e+06 disk.sdb1.in‚0 disk.sda.in‚ 3.62890625# disk.nvme0n1.in‚328984.25390625 network.sent‚ 1.3967341e+07 disk.dm-0.in‚529714.73046875 disk.sda1.in‚0 disk.sda2.out‚ 717.35546875 disk.sdb2.in‚ 3.49609375 disk.rbd0.in‚0.015625 disk.sdb.out‚ 717.35546875% memory_percent‚36.086225509643555 disk.sda2.in‚ 3.62890625 disk.md0.out‚ 691.7265625 disk.rbd2.in‚0) proc.memory.availableMB‚ 10471.6328125 cpu‚25.16548568715618 disk.sda1.out‚0 disk.rbd2.out‚ 1.62109375" disk.nvme1n1.in‚200730.4765625 disk.sdb2.out‚ 717.35546875" disk.nvme1n1.out‚ 46996.6171875 disk.sda.out‚ 717.35546875" disk.nvme0n1.out‚ 36666.0703125) proc.memory.percent‚18.49820613861084 proc.cpu.threads‚17 disk.sdb1.out‚0'&Ñ‚(:‡ ƒÊçÔ‘ˆÙ‡ gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚141.427' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.35675 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ä!­Ò‚(:ô ƒÊçÔŠªŸˆ% disk./.usageGB‚2975.0737838745117# disk.nvme0n1.out‚36666.14453125 disk.md0p1.in‚7.125 disk.rbd0.out‚ 1.68359375# proc.memory.rssMB‚ 3030.74609375 network.recv‚ 2.207828e+06" disk.nvme1n1.out‚ 47293.8984375# memory_percent‚36.0856294631958* proc.memory.availableMB‚10471.73046875) proc.memory.percent‚18.49820613861084 cpu‚25.165099051772522 network.sent‚ 1.4076454e+07# disk.nvme0n1.in‚328984.25390625 disk.dm-0.in‚529714.73046875 disk.sda2.in‚ 3.62890625! disk.sda2.out‚719.31396484375 disk.sdb1.in‚0 disk.sdb.out‚719.31396484375 disk.sda.in‚ 3.62890625 disk.dm-0.out‚83960.04296875 disk.md0p1.out‚ 693.62109375! disk.sdb2.out‚719.31396484375 disk.md0.in‚7.125 disk.rbd2.in‚0 disk.sda.out‚719.31396484375 proc.cpu.threads‚17) disk./.usagePercent‚83.18430816224415 disk.sda1.out‚0 disk.sdb2.in‚ 3.49609375" disk.nvme1n1.in‚200730.4765625 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sdb.in‚ 3.49609375 disk.md0.out‚ 693.62109375 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 1.62109375"º4T’Ó‚(:‡ ’ÊçÔíÛ‡ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.357' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.83925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07Ý5ñjÔ‚(:ß ’ÊçÔ¤©ø‡ network.recv‚ 2.21247e+06 disk.sdb.in‚ 3.49609375) proc.memory.percent‚18.49820613861084 cpu‚25.164765848019446 disk.dm-0.in‚529714.73046875 disk.sda1.out‚0 disk.md0p1.in‚7.125 disk.sdb2.in‚ 3.49609375 disk.md0.in‚7.125) disk./.usagePercent‚83.18429376307232 disk.rbd2.in‚0" disk.nvme1n1.in‚200730.4765625" disk.nvme1n1.out‚ 47472.0703125 disk.sda2.in‚ 3.62890625 disk.md0.out‚ 695.01171875$ disk./.usageGB‚2975.073268890381 disk.md0p1.out‚ 695.01171875 disk.sdb2.out‚ 720.76171875# proc.memory.rssMB‚ 3030.74609375 network.sent‚ 1.4080672e+07 disk.sdb1.out‚0# disk.nvme0n1.in‚328984.25390625 disk.dm-0.out‚ 84138.4453125( proc.memory.availableMB‚ 10471.484375 disk.sdb1.in‚0 disk.sda.out‚ 720.76171875 disk.nvme0n1.out‚ 36666.375 disk.sda2.out‚ 720.76171875 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 1.68359375$ memory_percent‚36.08713150024414 disk.sda.in‚ 3.62890625 proc.cpu.threads‚17 disk.rbd2.out‚ 1.62109375 disk.sdb.out‚ 720.76171875 disk.sda1.in‚0Î_Õf2Õ- _wandbruntime‚6614  _runtime‚6614Ë/;ÆÖÀ % _timestamp‚1.7884705609132123e+09 _runtime‚ 6614.09810425 _step‚464 & lr-AdamW/pg1‚0.0005239393964351427 & lr-AdamW/pg2‚0.0005239393964351427 trainer/global_step‚5699øÿ¿Ë×Å & lr-AdamW/pg1‚0.0005239393964351427 & lr-AdamW/pg2‚0.0005239393964351427 trainer/global_step‚5699 % _timestamp‚1.7884705609132123e+09 _runtime‚ 6614.09810425 _step‚464ÐL«™2Ø- _wandbruntime‚6614  _runtime‚6614¿ÄÝOÙèO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚673 % _timestamp‚1.7884705609141045e+09 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 108056255 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚231524694568960 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 trainer/global_step‚5699 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 108056255 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 154766908 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11314998891008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41023366 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 _step‚465 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11282930829934 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚256775224091648 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚245266914582016 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 262823161 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11314998891008 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11508333449728 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚220377675005264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11315232559616 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11315232559616 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚242839945131520 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 105546675 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚242839693459968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.ßÒœeŸallocated‚ 154793252 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚242839693459968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚231524712571904 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 262849507 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11283159811018 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 108056253 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 105547350 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64523984 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64523982 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚231524712571904 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚220377657965904 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚231660834816282Iò¼ó'ÚíO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚6614.099103534 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚256775536553984 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚242839693459968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚231524712571904 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 262849507 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11283159811018 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 108056253 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 105547350 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64523984 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64523982 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚231524712571904 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚220377657965904 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚231660834816282 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚673 % _timestamp‚1.7884705609141045e+09 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 108056255 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚231524694568960 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 trainer/global_step‚5699 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 108056255 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 154766908 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11314998891008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41023366 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 _step‚465 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11282930829934 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚256775224091648 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚245266914582016 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 262823161 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11314998891008 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11508333449728 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚220377675005264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11315232559616 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11315232559616 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚242839945131520 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 105546675 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚242839693459968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 154793252 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 262894500 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108074530 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108074530 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11510323420160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41029515 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚256819214675456 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11285118300212 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11284889318920 _step‚466 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚231700077642248 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11317196598784 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚256818898032128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 262868128 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108074528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 154793600 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11316962916864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 trainer/global_step‚5699 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 262894500 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11317196598784 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 154819970 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚64534784 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11510616123392 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚242880943522816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚669 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41030184 _runtime‚6615.050598451 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚245308574611968 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 154793600 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154819970 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11316962916864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 262868128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚671 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64534786 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105564299 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚242881195207680 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370"Ãf ßõM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41030184 _runtime‚6615.050598451 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚245308574611968 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 154793600 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154819970 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11316962916864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 262868128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚671 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64534786 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105564299 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚242881195207680 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 262894500 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108074530 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108074530 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11510323420160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitoÍ‚Kî r.on_train_batch_end/inactive_split.small_pool.freed‚41029515 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚256819214675456 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11285118300212 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11284889318920 _step‚466 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚231700077642248 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11317196598784 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚256818898032128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 262868128 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108074528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 154793600 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11316962916864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 trainer/global_step‚5699 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 262894500 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11317196598784 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 154819970 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚64534784 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11510616123392 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚242880943522816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚669 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496ÒD¯¯Ù2à- _wandbruntime‚6615  _runtime‚6615ßcXù]á× 0loss/FBt1/assignment_loss‚0.0801329003471551 .loss/FBt4/detection_loss‚2.440867191553116 0loss/FRt2/assignment_loss‚3.6021850719008337 2loss/FBt3/assignment_loss‚0.026084978080170726 /loss/FRt1/detection_loss‚2.9324088799101964 .loss/FRt2/detection_loss‚2.165946587725261 /loss/FRt4/detection_loss‚2.7551406468114545 /loss/FBt2/detection_loss‚2.3181760636242954 % _timestamp‚1.7884705618674164e+09 0loss/FRt4/assignment_loss‚3.2593842206462735 2loss/SRqqt3/assignment_loss‚0.8580775730661117 &loss/total_loss‚42.675657895841724 _runtime‚6615.052109787 .loss/FRt3/detection_loss‚1.647354933832373 0loss/SRqqt4/detection_loss‚2.293627987305323 2loss/SRqqt1/assignment_loss‚1.2163209979390275 2loss/SRqqt2/assignment_loss‚1.6360203085201128 1loss/FBt2/assignment_loss‚0.06445004233286124 1loss/FBt4/assignment_loss‚0.12671109545038917 epoch‚4 trainer/global_step‚5699 2loss/SRqqt4/assignment_loss‚1.4943519122898579 1loss/SRqqt1/detection_loss‚1.7928051218904297 0loss/SRqqt2/detection_loss‚2.372170578581946 1loss/SRqqt3/detection_loss‚1.0841219838766847 .loss/FBt1/detection_loss‚2.978466069698334 /loss/FBt3/detection_loss‚0.6982358539579053 _step‚467 0loss/FRt1/assignment_loss‚3.0870093532970975 /loss/FRt3/assignment_loss‚1.745607543204512gásÙbâÜ 1loss/SRqqt1/detection_loss‚1.7928051218904297 0loss/SRqqt2/detection_loss‚2.372170578581946 1loss/SRqqt3/detection_loss‚1.0841219838766847 .loss/FBt1/detection_loss‚2.978466069698334 /loss/FBt3/detection_loss‚0.6982358539579053 _step‚467 0loss/FRt1/assignment_loss‚3.0870093532970975 /loss/FRt3/assignment_loss‚1.745607543204512 0loss/FBt1/assignment_loss‚0.0801329003471551 .loss/FBt4/detection_loss‚2.440867191553116 0loss/FRt2/assignment_loss‚3.6021850719008337 2loss/FBt3/assignment_loss‚0.026084978080170726 /loss/FRt1/detection_loss‚2.9324088799101964 .loss/FRt2/detection_loss‚2.165946587725261 /loss/FRt4/detection_loss‚2.7551406468114545 /loss/FBt2/detection_loss‚2.3181760636242954 % _timestamp‚1.7884705618674164e+09 0loss/FRt4/assignment_loss‚3.2593842206462735 2loss/SRqqt3/assignment_loss‚0.8580775730661117 &loss/total_loss‚42.675657895841724 _runtime‚6615.052109787 .loss/FRt3/detection_loss‚1.647354933832373 0loss/SRqqt4/detection_loss‚2.293627987305323 2loss/SRqqt1/assignment_loss‚1.2163209979390275 2loss/SRqqt2/assignment_loss‚1.6360203085201128 1loss/FBt2/assignment_loss‚0.06445004233286124 1loss/FBt4/assignment_loss‚0.12671109545038917 epoch‚4 trainer/global_step‚5699 2loss/SRqqt4/assignment_loss‚1.4943519122898579Ók«ƒ3’ã‚(:‡ °ÊçÔá…ð‡ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚140.145' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.03625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿üFHwä‚(:ì °ÊçÔƒó©ˆ disk.md0.out‚ 698.40625$ memory_percent‚36.08736991882324 cpu‚25.16405839043331) disk./.usagePercent‚83.14708832959708% disk./.usageGB‚2973.7426223754883 disk.sdb1.out‚0 disk.sda.in‚ 3.62890625 disk.sda.out‚724.29052734375 disk.dm-0.out‚84576.77734375 disk.sda1.out‚0" disk.nvme0n1.out‚ 36666.5859375 disk.sda1.in‚0# disk.nvme1n1.out‚47910.19140625) proc.memory.availableMB‚ 10471.4453125 disk.md0p1.in‚7.125) proc.memory.percent‚18.49820613861084 network.recv‚ 2.22334e+06 disk.sdb1.in‚0# disk.nvme0n1.in‚328984.25390625 disk.rbd2.in‚0 disk.rbd2.out‚ 1.62109375 disk.sdb.in‚ 3.49609375# proc.memory.rssMB‚ 3030.74609375 disk.dm-0.in‚529714.73046875! disk.sda2.out‚724.29052734375 disk.md0.in‚7.125 disk.sda2.in‚ 3.62890625 network.sent‚ 1.4195541e+07 disk.md0p1.out‚ 698.40625 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 1.68359375" disk.nvme1n1.in‚200730.4765625 disk.sdb.out‚724.29052734375 proc.cpu.threads‚17 disk.sdb2.in‚ 3.49609375! disk.sdb2.out‚724.29052734375ÄÓKÀå‚(:… ¿ÊçÔ™ÐÃ… gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚139.528' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾ÑgMwæ‚(:ì ¿ÊçÔšåó†# disk.nvme1n1.out‚47910.91015625 disk.sdb.in‚ 3.49609375 disk.sda.in‚ 3.62890625 disk.sdb1.out‚0 network.sent‚ 1.4281255e+07( disk./.usagePercent‚83.1477378922377 disk.sda1.out‚0$ memory_percent‚36.08741760253906 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.sdb.out‚725.79931640625& proc.memory.availableMB‚ 10471.4375) proc.memory.percent‚18.49820613861084 proc.cpu.threads‚17$ disk./.usageGB‚2973.765853881836# disk.nvme0n1.in‚328984.25390625 disk.dm-0.out‚ 84577.5703125 disk.md0p1.out‚ 699.8515625! disk.sdb2.out‚725.79931640625 cpu‚25.16373856244378 disk.sda.out‚725.79931640625 disk.sda1.in‚0 disk.sda2.in‚ 3.62890625 disk.rbd2.out‚ 1.62109375 network.recv‚ 2.229303e+06# disk.nvme0n1.out‚36666.66015625 disk.md0p1.in‚7.125 disk.md0.in‚7.125! disk.sda2.out‚725.79931640625 disk.md0.out‚ 699.8515625 disk.sdb1.in‚0# proc.memory.rssMB‚ 3030.74609375 disk.dm-0.in‚529714.73046875 disk.sdb2.in‚ 3.49609375 disk.rbd0.out‚ 1.68359375" disk.nvme1n1.in‚200730.4765625ïOઑç‚(:† ÎÊçԇʾ† gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚138.642' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.6605 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n¬àxè‚(:í ÎÊçԌܧ‡" disk.nvme1n1.in‚200730.4765625 network.recv‚ 2.233814e+06 disk.sdb2.in‚ 3.49609375 network.sent‚ 1.4285396e+07" disk.nvme1n1.out‚ 47981.7421875 disk.sda2.in‚ 3.62890625 disk.rbd2.in‚0 disk.md0p1.out‚ 701.8828125) proc.memory.percent‚18.49820613861084( disk./.usagePercent‚83.1454338114228$ disk./.usageGB‚2973.683448791504 disk.rbd2.out‚ 1.62109375 disk.sda.out‚727.89697265625# disk.nvme0n1.in‚328984.25390625 disk.dm-0.in‚529714.73046875 disk.sda1.out‚0! disk.sdb2.out‚727.89697265625 disk.sda.in‚ 3.62890625 disk.dm-0.out‚84649.14453125 disk.md0p1.in‚7.125$ memory_percent‚36.09046936035156# proc.memory.rssMB‚ 3030.74609375 disk.sdb1.in‚0 disk.sdb1.out‚0# disk.nvme0n1.out‚36667.40234375 disk.sda1.in‚0! disk.sda2.out‚727.89697265625& proc.memory.availableMB‚ 10470.9375 proc.cpu.threads‚17 disk.md0.in‚7.125 disk.md0.out‚ 701.8828125 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 1.68359375 disk.sdb.in‚ 3.49609375 disk.sdb.out‚727.89697265625 cpu‚25.163405780490137Öý¶Ñ2é- _wandbruntime‚6665  _runtime‚6665üãøÇêÁ trainer/global_step‚5749 % _timestamp‚1.7884706123423784e+09 _runtime‚6665.527265385 _step‚468 & lr-AdamW/pg1‚0.0005181996854128676 & lr-AdamW/pg2‚0.0005181996854128676tÀÌÌëÆ % _timestamp‚1.7884706123423784e+09 _runtime‚6665.527265385 _step‚468 & lr-AdamW/pg1‚0.0005181996854128676 & lr-AdamW/pg2‚0.0005181996854128676 trainer/global_step‚5749ÔJd2ì- _wandbruntime‚6665  _runtime‚6665†:ߣî'íèO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 106433995 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 % _timestamp‚1.7884706123433213e+09 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚244902462687744 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚258959638321664 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚247351113084416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 156131204 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚233488996418560 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41369226 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚233635044707546 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚244902211016192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _step‚469 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚233489014421504 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 156104860 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11608240909312 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 265074863 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11413214597632 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 265101209 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚244902462687744 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚258959638321664 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚247351113084416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 156131204 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚233488996418560 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41369226 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚233635044707546 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚244902211016192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _step‚469 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚233489014421504 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 156104860 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11608240909312 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 265074863 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11413214597632 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 265101209 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚247392797054464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108988280 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11415178623488 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11415412305408 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚233528300458496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11610230784000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 % _timestamp‚1.7884706132862232e+09 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106451887 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚247392773114368 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚244943712763904 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚678 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚244943712763904 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41376325 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚233674779574608 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚222291723233888 _runtime‚6666.471699696 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 156131552 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 265146202 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11415412305408 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚5749 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 156131552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65074886 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚676 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11383056340720 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11382827359428 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚233528282455552 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106451209 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚259003003898368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11610525584384 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265146202 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 265119830 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1‡Ø¦!ý&ñ÷M QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚247392773114368 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚244943712763904 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚678 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚244943712763904 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41376325 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚233674779574608 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚222291723233888 _runtime‚6666.471699696 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 156131552 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 265146202 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11415412305408 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚5749 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 156131552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65074886 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚676 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11383056340720 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11382827359428 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚233528282455552 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106451209 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚259003003898368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11610525584384 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265146202 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 265119830 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65074884 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚259003322638848 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚247392797054464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108988280 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11415178623488 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11415412305408 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚233528300458496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11610230784000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 % _timestamp‚1.7884706132862232e+09 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106451887Ö±'×”2ò- _wandbruntime‚6666  _runtime‚6666žJÎ;[óÕ 1loss/SRqqt4/assignment_loss‚1.525320924197634 0loss/FBt4/assignment_loss‚0.1589108920933794 /loss/FRt2/assignment_loss‚3.191861607926957 2loss/SRqqt3/assignment_loss‚0.9264532149690821 /loss/FBt2/detection_loss‚2.3034368289841547 epoch‚4 2loss/SRqqt2/assignment_loss‚1.6881984774813508 2loss/FBt1/assignment_loss‚0.014048070362251665 1loss/SRqqt3/detection_loss‚1.0089672322732857 /loss/FBt3/detection_loss‚0.4629980087798985 .loss/FBt4/detection_loss‚1.841879970752276 1loss/FBt3/assignment_loss‚0.05885125171808835 /loss/FRt1/detection_loss‚2.6894077537581325 /loss/FRt3/detection_loss‚1.6320927395317397 1loss/SRqqt1/detection_loss‚2.0760303402230855 0loss/FRt3/assignment_loss‚1.3125455613996162 1loss/SRqqt4/detection_loss‚2.5606721977392835 trainer/global_step‚5749 % _timestamp‚1.7884706132886624e+09 _runtime‚6666.473370394 /loss/FRt4/assignment_loss‚3.071718977557288 2loss/SRqqt1/assignment_loss‚1.3662764015164048 .loss/FRt4/detection_loss‚2.695297492047151 .loss/FBt1/detection_loss‚2.766141732533773 /loss/FRt1/assignment_loss‚2.924928231164813 1loss/FBt2/assignment_loss‚0.21461477466962403 .loss/FRt2/detection_loss‚2.158761997930111 1loss/SRqqt2/detection_loss‚2.4714265303178267 %loss/total_loss‚41.12084120992721 _step‚471!ãŠì`ôÚ _step‚471 1loss/SRqqt4/assignment_loss‚1.525320924197634 0loss/FBt4/assignment_loss‚0.1589108920933794 /loss/FRt2/assignment_loss‚3.191861607926957 2loss/SRqqt3/assignment_loss‚0.9264532149690821 /loss/FBt2/detection_loss‚2.3034368289841547 epoch‚4 2loss/SRqqt2/assignment_loss‚1.6881984774813508 2loss/FBt1/assignment_loss‚0.014048070362251665 1loss/SRqqt3/detection_loss‚1.0089672322732857 /loss/FBt3/detection_loss‚0.4629980087798985 .loss/FBt4/detection_loss‚1.841879970752276 1loss/FBt3/assignment_loss‚0.05885125171808835 /loss/FRt1/detection_loss‚2.6894077537581325 /loss/FRt3/detection_loss‚1.6320927395317397 1loss/SRqqt1/detection_loss‚2.0760303402230855 0loss/FRt3/assignment_loss‚1.3125455613996162 1loss/SRqqt4/detection_loss‚2.5606721977392835 trainer/global_step‚5749 % _timestamp‚1.7884706132886624e+09 _runtime‚6666.473370394 /loss/FRt4/assignment_loss‚3.071718977557288 2loss/SRqqt1/assignment_loss‚1.3662764015164048 .loss/FRt4/detection_loss‚2.695297492047151 .loss/FBt1/detection_loss‚2.766141732533773 /loss/FRt1/assignment_loss‚2.924928231164813 1loss/FBt2/assignment_loss‚0.21461477466962403 .loss/FRt2/detection_loss‚2.158761997930111 1loss/SRqqt2/detection_loss‚2.4714265303178267 %loss/total_loss‚41.12084120992721×ÂÛþõ‚(:… ÝÊçÔò³¬† gpu.0.gpu‚39.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚143.872' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.968 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.î$#tö‚(:é ÝÊçÔÐà† disk.sdb2.in‚ 3.49609375 disk.md0.in‚7.125 disk.sdb1.in‚0 disk.sda2.in‚ 3.62890625$ disk./.usageGB‚2973.986488342285 disk.rbd0.out‚ 1.6953125" disk.nvme1n1.in‚200730.4765625 disk.dm-0.out‚ 84864.4375$ memory_percent‚36.08906269073486 disk.md0.out‚ 703.27734375 disk.rbd2.out‚ 1.6953125 disk.sdb1.out‚0 disk.sda.out‚729.3525390625 disk.sdb2.out‚729.3525390625 disk.sdb.in‚ 3.49609375 disk.sdb.out‚729.3525390625 disk.md0p1.out‚ 703.27734375# proc.memory.rssMB‚ 3030.74609375 disk.dm-0.in‚529714.73046875 disk.sda2.out‚729.3525390625* proc.memory.availableMB‚10471.16796875" disk.nvme1n1.out‚ 48111.0390625 network.sent‚ 1.4291203e+07 network.recv‚ 2.237986e+06 disk.rbd2.in‚0) disk./.usagePercent‚83.15390692409457 disk.sda.in‚ 3.62890625# disk.nvme0n1.in‚328984.25390625" disk.nvme0n1.out‚ 36753.3984375 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0p1.in‚7.125 disk.rbd0.in‚0.015625) proc.memory.percent‚18.49820613861084 cpu‚25.163119324636344 proc.cpu.threads‚17;õ í’÷‚(:‡ ìÊçÔò¢Õ‡ gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚119.701' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.92525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 @Vªvø‚(:ë ìÊçÔÍ㔈 network.recv‚ 2.245034e+06 disk.sda1.in‚0 disk.rbd2.in‚0$ memory_percent‚36.08837127685547# proc.memory.rssMB‚ 3030.74609375 proc.cpu.threads‚17 disk.md0.in‚7.125 disk.md0p1.out‚ 705.30078125 disk.sdb2.out‚731.4365234375' proc.memory.availableMB‚ 10471.28125 disk.sda1.out‚0 disk.md0.out‚ 705.30078125 disk.rbd2.out‚ 1.6953125# disk.nvme1n1.out‚48199.33984375 network.sent‚ 1.4401003e+07 disk.sda2.out‚731.4365234375 disk.md0p1.in‚7.125 disk.rbd0.out‚ 1.7578125" disk.nvme1n1.in‚200730.4765625 disk.sdb.out‚731.4365234375 disk.sda.in‚ 3.62890625# disk.nvme0n1.in‚328984.25390625) disk./.usagePercent‚83.15623724340035 disk.dm-0.out‚ 85038.796875 cpu‚25.162818755131113 disk.sda2.in‚ 3.62890625 disk.sdb1.out‚0 disk.sda.out‚731.4365234375 disk.dm-0.in‚529714.73046875) proc.memory.percent‚18.49820613861084 disk.sdb1.in‚0# disk.nvme0n1.out‚36839.45703125% disk./.usageGB‚2974.0698318481445 disk.sdb2.in‚ 3.49609375 disk.rbd0.in‚0.015625 disk.sdb.in‚ 3.49609375„»°’ù‚(:‡ ûÊçÔô›â‡ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚137.083' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.27075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ZDßÖú‚(:ô ûÊçÔ§ËΈ# disk.nvme1n1.out‚48282.48046875 disk.sdb1.out‚0 disk.dm-0.in‚529714.73046875( disk./.usagePercent‚83.1563888080165# disk.nvme0n1.in‚328984.25390625 cpu‚25.162527061304523 network.sent‚ 1.4404092e+07 disk.sdb2.in‚ 3.49609375! disk.sdb2.out‚732.84423828125 disk.rbd0.in‚0.015625 disk.rbd2.in‚0! disk.sda2.out‚732.84423828125 disk.md0p1.out‚ 706.6328125 disk.md0.in‚7.125" disk.nvme0n1.out‚ 36845.1484375 disk.dm-0.out‚85127.62890625 disk.sda1.out‚0% memory_percent‚36.089158058166504 network.recv‚ 2.248521e+06 disk.md0.out‚ 706.6328125 disk.sdb.in‚ 3.49609375# proc.memory.rssMB‚ 3030.74609375) proc.memory.percent‚18.49820613861084$ disk./.usageGB‚2974.075252532959 disk.sda2.in‚ 3.62890625 disk.sdb1.in‚0 disk.md0p1.in‚7.125 disk.rbd2.out‚ 1.79736328125" disk.nvme1n1.in‚200730.4765625 disk.sdb.out‚732.84423828125 disk.sda.in‚ 3.62890625 disk.sda.out‚732.84423828125* proc.memory.availableMB‚10471.15234375 disk.sda1.in‚0 proc.cpu.threads‚17 disk.rbd0.out‚ 1.7578125df2û- _wandbruntime‚6719  _runtime‚6719õe…^ÇüÁ _step‚472 & lr-AdamW/pg1‚0.0005124378897065526 & lr-AdamW/pg2‚0.0005124378897065526 trainer/global_step‚5799 % _timestamp‚1.7884706665582328e+09 _runtime‚6719.743114819ž$GÌýÆ & lr-AdamW/pg2‚0.0005124378897065526 trainer/global_step‚5799 % _timestamp‚1.7884706665582328e+09 _runtime‚6719.743114819 _step‚472 & lr-AdamW/pg1‚0.0005124378897065526Øø¯¹2þ- _wandbruntime‚6719  _runtime‚6719¨E'ÿèO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 267326565 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11511663972864 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚235609746639698 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 157442812 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚224130710747664 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚246964980243968 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 267352911 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11708393352192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚473 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚249435287646720 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 109883753 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65604184 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚261143704939008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚235453298268160 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 157469156 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚673 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚235609500619254 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11478806910950 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚246964980243968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109883753 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41715796 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚224130693708304 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109883755 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚675 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 299008000 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 VHDeviceStatsMonitor.on_trainú¢Ô_batch_start/inactive_split_bytes.all.current‚ 322948096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11511430304256 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 107319305 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚6719.744277734 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65604182 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚246964728572416 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚249435311586816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚235453316271104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚235453298268160 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚246964728572416 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107319980 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚261143381990912 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11479035892034 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚5799 id¦ó'€íO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 157442812 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11511663972864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11708393352192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚473 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚249435287646720 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 109883753 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65604184 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚261143704939008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚235453298268160 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 157469156 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚673 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚235609500619254 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11478806910950 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚246964980243968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109883753 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41715796 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚224130693708304 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109883755 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚675 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 299008000 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 322948096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11511430304256 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 107319305 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚6719.744277734 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65604182 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚246964728572416 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚249435311586816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚235453316271104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚235453298268160 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚246964728572416 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107319980 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚261143381990912 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11479035892034 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚5799 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 267326565 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11511663972864 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚235609746639698 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 157442812 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚224130710747664 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚246964980243968 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 267352911 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11513628012032 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 267371532 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚224168241105088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 trainer/global_step‚5799 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 _runtime‚6720.776362254 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11513394330112 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚681 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65614986 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157495874 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚235649235486316 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 267371532 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11513394330112 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚247005978635264 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 267397904 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚261187062796800 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11480765399936 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚235492584305152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 157469504 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 157469504 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚235492602308096 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107336638 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚249476995556864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚235492602308096 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11513628012032 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 107337321 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41722335 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚249476971616768 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11480994381228 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109902030 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109902028 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚235648989465664vóy¡Ï…÷M PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 267397904 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚261187062796800 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11480765399936 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚235492584305152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 157469504 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 157469504 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚235492602308096 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107336638 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚249476995556864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚235492602308096 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11513628012032 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 107337321 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41722335 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚249476971616768 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11480994381228 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109902030 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109902028 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚235648989465664 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11513628012032 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 267371532 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚224168241105088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 trainer/global_step‚5799 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 _runtime‚6720.776362254 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11513394330112 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚681 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2£šï€.73 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65614986 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157495874 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚235649235486316 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 267371532 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11513394330112 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚247005978635264 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ÚsùÓ2†- _wandbruntime‚6720  _runtime‚6720C¯l—Z‡Ô 2loss/SRqqt2/assignment_loss‚1.8473782029934227 2loss/SRqqt3/assignment_loss‚0.7981931021703141 1loss/FBt4/assignment_loss‚0.23833810397060992 .loss/FRt1/detection_loss‚2.745547654107213 /loss/FBt4/detection_loss‚2.2324248353640237 /loss/FRt4/assignment_loss‚2.892642687106955 .loss/FRt2/detection_loss‚2.227309677020336 1loss/SRqqt3/detection_loss‚0.9688632600896415 /loss/FBt2/detection_loss‚1.9959575556791747 _step‚475 /loss/FRt2/assignment_loss‚3.070402998394436 0loss/FBt2/assignment_loss‚0.1592780928297613 1loss/FBt3/assignment_loss‚0.01647652197465277 /loss/FBt1/detection_loss‚2.6295801315988814 _runtime‚ 6720.77757661 /loss/FBt1/assignment_loss‚0.398960770969422 /loss/FRt3/detection_loss‚1.3484013962614185 1loss/SRqqt1/detection_loss‚1.6061723147119795 0loss/SRqqt4/detection_loss‚2.371051595066533 0loss/FRt1/assignment_loss‚2.2231697905808687 2loss/SRqqt4/assignment_loss‚1.5391313200534293 /loss/FRt4/detection_loss‚2.9322001358558394 /loss/FBt3/detection_loss‚0.8570252508423957 epoch‚4 0loss/FRt3/assignment_loss‚1.4519751054399155 2loss/SRqqt1/assignment_loss‚1.4271421054052928 1loss/SRqqt2/detection_loss‚2.5074503771960734 %loss/total_loss‚40.48507298568259 trainer/global_step‚5799 % _timestamp‚1.7884706675927563e+09”rä2_ˆÙ _step‚475 /loss/FRt2/assignment_loss‚3.070402998394436 0loss/FBt2/assignment_loss‚0.1592780928297613 1loss/FBt3/assignment_loss‚0.01647652197465277 /loss/FBt1/detection_loss‚2.6295801315988814 _runtime‚ 6720.77757661 /loss/FBt1/assignment_loss‚0.398960770969422 /loss/FRt3/detection_loss‚1.3484013962614185 1loss/SRqqt1/detection_loss‚1.6061723147119795 0loss/SRqqt4/detection_loss‚2.371051595066533 0loss/FRt1/assignment_loss‚2.2231697905808687 2loss/SRqqt4/assignment_loss‚1.5391313200534293 /loss/FRt4/detection_loss‚2.9322001358558394 /loss/FBt3/detection_loss‚0.8570252508423957 epoch‚4 0loss/FRt3/assignment_loss‚1.4519751054399155 2loss/SRqqt1/assignment_loss‚1.4271421054052928 1loss/SRqqt2/detection_loss‚2.5074503771960734 %loss/total_loss‚40.48507298568259 trainer/global_step‚5799 % _timestamp‚1.7884706675927563e+09 2loss/SRqqt2/assignment_loss‚1.8473782029934227 2loss/SRqqt3/assignment_loss‚0.7981931021703141 1loss/FBt4/assignment_loss‚0.23833810397060992 .loss/FRt1/detection_loss‚2.745547654107213 /loss/FBt4/detection_loss‚2.2324248353640237 /loss/FRt4/assignment_loss‚2.892642687106955 .loss/FRt2/detection_loss‚2.227309677020336 1loss/SRqqt3/detection_loss‚0.9688632600896415 /loss/FBt2/detection_loss‚1.9959575556791747ÛÐE ‰‚(:… ™ËçÔóλˆ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚141.704' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.426 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gÒû0uŠ‚(:ê ™ËçÔ’‹Ýˆ disk.sda.out‚736.3095703125! disk.nvme0n1.in‚ 329292.546875 disk.dm-0.in‚ 530211.390625 disk.sda1.out‚0% memory_percent‚36.089491844177246 network.sent‚ 1.4519282e+07# disk.nvme0n1.out‚37178.35546875 disk.sda2.out‚736.3095703125 disk.rbd0.in‚0.015625* proc.memory.availableMB‚10471.09765625# disk.nvme1n1.out‚48459.73828125 disk.sda.in‚ 3.62890625 disk.sdb2.in‚ 3.49609375 disk.rbd2.out‚ 1.8369140625 disk.nvme1n1.in‚ 200918.84375 disk.sdb1.in‚0 disk.rbd0.out‚ 1.8369140625# proc.memory.rssMB‚ 3030.74609375 disk.md0p1.in‚7.125 disk.sdb2.out‚736.3095703125 disk.dm-0.out‚ 85638.09375 disk.sdb.in‚ 3.49609375) proc.memory.percent‚18.49820613861084 network.recv‚ 2.259944e+06) disk./.usagePercent‚83.16145294342111 disk.sda2.in‚ 3.62890625 disk.md0p1.out‚ 709.953125 disk.md0.in‚7.125 disk.md0.out‚ 709.953125 disk.rbd2.in‚0 cpu‚25.161953518282047% disk./.usageGB‚2974.2563705444336 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sdb.out‚736.3095703125 proc.cpu.threads‚17¥žP‘‹‚(:† ¨ËçÔíŒÑ‡ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.357' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.83925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èóÎtŒ‚(:é ¨ËçԠ錉 disk.sdb.in‚ 3.49609375 network.recv‚ 2.26655e+06 disk.md0p1.in‚7.125 disk.md0p1.out‚ 711.4765625 disk.rbd0.out‚ 1.89990234375 disk.sdb1.in‚0 disk.sdb1.out‚0! disk.sda2.out‚737.88623046875% memory_percent‚36.091041564941406 disk.md0.out‚ 711.4765625' proc.memory.availableMB‚ 10470.84375 cpu‚25.16166620617653 proc.cpu.threads‚17! disk.nvme1n1.out‚ 48528.578125 disk.sda2.in‚ 3.62890625 disk.dm-0.out‚ 85906.6484375) disk./.usagePercent‚83.17736210841079% disk./.usageGB‚2974.8253593444824 disk.sdb2.in‚ 3.49609375 disk.rbd2.in‚0 disk.rbd2.out‚ 1.8369140625 disk.sda.out‚737.88623046875" disk.nvme0n1.out‚ 37378.0703125! disk.nvme0n1.in‚ 329292.546875 disk.dm-0.in‚ 530211.390625 disk.nvme1n1.in‚ 200918.84375 disk.sda.in‚ 3.62890625) proc.memory.percent‚18.49820613861084! disk.sdb2.out‚737.88623046875 disk.md0.in‚7.125 disk.rbd0.in‚0.015625 disk.sdb.out‚737.88623046875 disk.sda1.in‚0# proc.memory.rssMB‚ 3030.74609375 disk.sda1.out‚0 network.sent‚ 1.46057e+07ϵ ã‘‚(:† ·ËçÔ¾´ø† gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚140.478' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.1195 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04ð܃`Ž‚(:Õ ·ËçԢч cpu‚25.161394311054785 network.sent‚ 1.4608902e+07 disk.rbd2.out‚ 1.89990234375 disk.sda.out‚739.625 disk.md0p1.out‚ 713.15625* proc.memory.availableMB‚10470.84765625# proc.memory.rssMB‚ 3030.74609375 disk.rbd2.in‚0 disk.sdb.in‚ 3.49609375 disk.sdb.out‚739.625! disk.nvme0n1.out‚ 37378.140625 disk.md0.in‚7.125) proc.memory.percent‚18.49820613861084 disk.dm-0.in‚530492.12890625 disk.dm-0.out‚85959.73046875 disk.sda1.out‚0 disk.sda2.out‚739.625 disk.sdb2.out‚739.625% memory_percent‚36.091017723083496 proc.cpu.threads‚17 network.recv‚ 2.270505e+06) disk./.usagePercent‚83.18086452030407 disk.sda2.in‚ 3.62890625# disk.nvme1n1.out‚48581.58984375 disk.sdb1.out‚0% disk./.usageGB‚2974.9506225585938# disk.nvme1n1.in‚200961.91796875 disk.md0.out‚ 713.15625 disk.md0p1.in‚7.125 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 1.89990234375 disk.sdb1.in‚0 disk.sda.in‚ 3.62890625" disk.nvme0n1.in‚329530.2109375 disk.sda1.in‚0 disk.sdb2.in‚ 3.49609375ŽìØ2- _wandbruntime‚6775  _runtime‚6775 ÕºÆÀ trainer/global_step‚5849 $ _timestamp‚1.788470721887979e+09 _runtime‚6775.072856622 _step‚476 & lr-AdamW/pg1‚0.0005066552100657867 & lr-AdamW/pg2‚0.0005066552100657867l;ÿË‘Å _step‚476 & lr-AdamW/pg1‚0.0005066552100657867 & lr-AdamW/pg2‚0.0005066552100657867 trainer/global_step‚5849 $ _timestamp‚1.788470721887979e+09 _runtime‚6775.072856622ÜqÄ+g2’- _wandbruntime‚6775  _runtime‚6775³,Ûbî'“èO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚226007211579504 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110797505 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 269604613 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚237584202551406 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚249027497800192 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚249027246128640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 158807108 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚237417618120704 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚237417600117760 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 158780764 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42061826 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚477 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110797503 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11576744951458 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚237417618120704 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11808314839552 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11609879679488 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 269578267 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 158807108 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66144284 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11808024220160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚237583956530962 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 % _timestamp‚1.7884707218889372e+09 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110797503 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _runtime‚6775.073939217 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11609646010880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚226007228618864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚251519510089216rµ/&ó'”íO RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 108206110 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66144282 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚263327824928768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 108205401 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚707 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110797505 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚249027246128640 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚263327510369280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11609646010880 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚237417600117760 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚251519486149120 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11609646010880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚226007228618864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚251519510089216 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚226007211579504 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110797505 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 269604613 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚237584202551406 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚249027497800192 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚249027246128640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 158807108 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚237417618120704 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚237417600117760 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 158780764 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42061826 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚477 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110797503 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11576744951458 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚237417618120704 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11808314839552 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11609879679488 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 269578267 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 158807108 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66144284 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11808024220160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚237583956530962 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 % _timestamp‚1.7884707218889372e+09 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110797503 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _runtime‚6775.073939217 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 K=DeviceStats ÍÁe’%Monitor.on_train_batch_end/active.small_pool.freed‚ 158807456 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚226044758976288 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11810309255680 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11611610036736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110815780 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11578703440444 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚249068747876352 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11611610036736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚251561170119168 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42068459 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 110815778 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 322934784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚237456886154752 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66155086 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 298994688 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚226044741936928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66155084 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚696 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11810010260992 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110815780 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚5849 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚237456904157696 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 108223545 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _step‚478 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 158807456 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚698 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 _runtime‚6776.136239103 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚237623691398024 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚237456886154752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11578932421736 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 158833826 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚251561194059264 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42067763 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚263371180380160 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚237623445377372 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚237456904157696 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11611843718656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108222847 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11611843718656 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 269649606 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136ÏR·gý&—÷M VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66155084 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚696 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11810010260992 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110815780 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚5849 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚237456904157696 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 108223545 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _step‚478 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 158807456 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚698 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 _runtime‚6776.136239103 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚237623691398024 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚237456886154752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11578932421736 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 158833826 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚251561194059264 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42067763 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚263371180380160 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚237623445377372 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚237456904157696 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11611843718656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108222847 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11611843718656 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 269649606 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 269623234 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 158807456 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚226044758976288 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11810309255680 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11611610036736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110815780 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11578703440444 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚249068747876352 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11611610036736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚251561170119168 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42068459 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 110815778 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 322934784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚237456886154752 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66155086 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 298994688 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚226044741936928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0Þxí­52˜- _wandbruntime‚6776  _runtime‚6776øØR™\™Ö /loss/FBt3/detection_loss‚0.6529197999371741 2loss/SRqqt4/assignment_loss‚1.5703847599881036 /loss/FRt2/detection_loss‚1.8771261310310297 /loss/FRt3/detection_loss‚1.5253733991994523 /loss/FRt1/assignment_loss‚2.823995921346876 /loss/FRt4/assignment_loss‚3.897043433454302 1loss/FBt4/assignment_loss‚0.14865572084318046 .loss/FRt4/detection_loss‚2.807754593866843 %loss/total_loss‚41.98570487789599 2loss/SRqqt1/assignment_loss‚1.6043488196200795 1loss/SRqqt3/assignment_loss‚0.778728740120476 0loss/SRqqt1/detection_loss‚2.018318102757136 -loss/FBt4/detection_loss‚2.22452574968338 epoch‚4 trainer/global_step‚5849 0loss/FRt3/assignment_loss‚1.5759225860238075 1loss/FBt2/assignment_loss‚0.09377050097884876 1loss/SRqqt2/detection_loss‚2.1561387741204463 1loss/SRqqt3/detection_loss‚0.8926921873353422 _runtime‚6776.137287988 _step‚479 0loss/FRt2/assignment_loss‚3.2617073430205292 2loss/SRqqt2/assignment_loss‚1.6907156039819573 % _timestamp‚1.7884707229525573e+09 /loss/FRt1/detection_loss‚2.7022225906451545 .loss/FBt1/detection_loss‚2.588139694929123 .loss/FBt2/detection_loss‚2.436565089970827 1loss/FBt1/assignment_loss‚0.14934980421071486 3loss/FBt3/assignment_loss‚0.0037677931381013287 1loss/SRqqt4/detection_loss‚2.5055377376931056¼ÊA˜ašÛ 1loss/SRqqt2/detection_loss‚2.1561387741204463 1loss/SRqqt3/detection_loss‚0.8926921873353422 _runtime‚6776.137287988 _step‚479 0loss/FRt2/assignment_loss‚3.2617073430205292 2loss/SRqqt2/assignment_loss‚1.6907156039819573 % _timestamp‚1.7884707229525573e+09 /loss/FRt1/detection_loss‚2.7022225906451545 .loss/FBt1/detection_loss‚2.588139694929123 .loss/FBt2/detection_loss‚2.436565089970827 1loss/FBt1/assignment_loss‚0.14934980421071486 3loss/FBt3/assignment_loss‚0.0037677931381013287 1loss/SRqqt4/detection_loss‚2.5055377376931056 /loss/FBt3/detection_loss‚0.6529197999371741 2loss/SRqqt4/assignment_loss‚1.5703847599881036 /loss/FRt2/detection_loss‚1.8771261310310297 /loss/FRt3/detection_loss‚1.5253733991994523 /loss/FRt1/assignment_loss‚2.823995921346876 /loss/FRt4/assignment_loss‚3.897043433454302 1loss/FBt4/assignment_loss‚0.14865572084318046 .loss/FRt4/detection_loss‚2.807754593866843 %loss/total_loss‚41.98570487789599 2loss/SRqqt1/assignment_loss‚1.6043488196200795 1loss/SRqqt3/assignment_loss‚0.778728740120476 0loss/SRqqt1/detection_loss‚2.018318102757136 -loss/FBt4/detection_loss‚2.22452574968338 epoch‚4 trainer/global_step‚5849 0loss/FRt3/assignment_loss‚1.5759225860238075 1loss/FBt2/assignment_loss‚0.09377050097884876ß7;.‘›‚(:† ÆËçÔâ÷‡ gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚133.958' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.4895 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v7|yœ‚(:î ÆËçÔ£à‰‰ disk.sdb2.out‚ 740.951171875 disk.rbd0.out‚ 1.962890625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda2.out‚ 740.951171875 disk.md0p1.out‚ 714.41796875# memory_percent‚36.0912561416626 network.recv‚ 2.274168e+06 disk.sdb.in‚ 3.49609375 disk.sda.out‚ 740.951171875 cpu‚25.161101265270005 proc.cpu.threads‚17 disk.sdb.out‚ 740.951171875 disk.sda1.out‚0 disk.md0.out‚ 714.41796875 disk.rbd0.in‚0.015625) disk./.usagePercent‚83.18434943987009 disk.md0.in‚7.125" disk.nvme1n1.in‚201032.6953125 disk.nvme1n1.out‚ 48734.46875) proc.memory.percent‚18.49820613861084% disk./.usageGB‚2975.0752601623535# disk.nvme0n1.in‚329679.39453125 disk.dm-0.in‚530712.08984375 disk.dm-0.out‚86312.37890625 disk.sda1.in‚0 disk.sda2.in‚ 3.62890625 disk.md0p1.in‚7.125 network.sent‚ 1.4627003e+07 disk.sda.in‚ 3.62890625* proc.memory.availableMB‚10470.80859375# disk.nvme0n1.out‚37577.91015625 disk.rbd2.in‚0 disk.rbd2.out‚ 1.89990234375 disk.sdb2.in‚ 3.49609375# proc.memory.rssMB‚ 3030.74609375ÕÎʼnœ‚(:‘ ÕËçÔ禋‡ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë›…~ž‚(:ó ÕËçÔê›Þ‡ cpu‚25.16086750029917 network.recv‚ 2.281213e+06 disk.sda1.out‚0! disk.sda2.out‚742.86083984375 disk.md0.out‚ 716.2578125 disk.dm-0.in‚530712.08984375 disk.rbd2.in‚0# disk.nvme1n1.out‚48861.62890625$ memory_percent‚36.08975410461426 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.sda.in‚ 3.62890625 disk.sda.out‚742.86083984375 disk.sda2.in‚ 3.62890625 disk.md0p1.in‚7.125 disk.sdb2.in‚ 3.49609375 disk.rbd0.out‚ 1.962890625" disk.nvme1n1.in‚201032.6953125 disk.sdb.out‚742.86083984375" disk.nvme0n1.out‚ 37910.3359375 network.sent‚ 1.4723706e+07) disk./.usagePercent‚83.18435103977806 disk.sdb1.in‚0! disk.sdb2.out‚742.86083984375# proc.memory.rssMB‚ 3030.74609375% disk./.usageGB‚2975.0753173828125 disk.sda1.in‚0 disk.md0.in‚7.125 disk.md0p1.out‚ 716.2578125) proc.memory.availableMB‚ 10471.0546875) proc.memory.percent‚18.49820613861084 disk.sdb.in‚ 3.49609375# disk.nvme0n1.in‚329679.39453125 disk.dm-0.out‚86771.96484375 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 1.962890625 ®â!‘Ÿ‚(:† äËçÔàÈ¿‡ gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚127.318' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.8295 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qìò{ ‚(:ð äËçÔÆý㇠network.recv‚ 2.285538e+06 disk.rbd0.in‚0.015625# disk.nvme1n1.out‚48861.93359375) proc.memory.percent‚18.49820613861084 disk.sda2.out‚744.3369140625 disk.rbd0.out‚ 1.962890625 disk.sdb.in‚ 4.48828125 disk.sda.out‚744.3369140625# disk.nvme0n1.in‚329776.41796875 network.sent‚ 1.4727481e+07$ disk./.usageGB‚2974.800552368164 disk.sdb2.in‚ 4.48828125 disk.md0.out‚ 717.6796875( disk./.usagePercent‚83.1766684949704 disk.sda1.in‚0 disk.md0p1.out‚ 717.6796875 disk.rbd2.out‚ 1.962890625# disk.nvme1n1.in‚201077.44140625 disk.sdb1.in‚0 disk.sda.in‚ 4.40234375 disk.dm-0.in‚ 530853.859375 disk.sdb2.out‚744.3369140625 disk.nvme0n1.out‚ 37910.65625 disk.dm-0.out‚86772.58984375 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sdb1.out‚0$ memory_percent‚36.08965873718262# proc.memory.rssMB‚ 3030.74609375 cpu‚25.16058761773791 disk.sda1.out‚0 disk.sda2.in‚ 4.40234375 disk.md0p1.in‚8.890625 disk.md0.in‚8.890625 disk.sdb.out‚744.3369140625) proc.memory.availableMB‚ 10471.0703125u¦ ’¡‚(:‡ óËçÔ⤆ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚137.693' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.42325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•ø|x¢‚(:í óËçÔò脇# disk.nvme1n1.in‚201077.44140625# disk.nvme1n1.out‚48904.87890625) proc.memory.availableMB‚ 10471.0546875 disk.rbd2.in‚0 disk.sdb1.in‚0 proc.cpu.threads‚17 disk.dm-0.out‚87081.98828125) proc.memory.percent‚18.49820613861084 cpu‚25.1603514668863 disk.md0.out‚ 719.140625 disk.sda.in‚ 4.40234375# disk.nvme0n1.in‚329776.41796875 disk.sda1.out‚0 disk.sda2.in‚ 4.40234375 disk.sda2.out‚745.8623046875 disk.md0p1.in‚8.890625 disk.md0.in‚8.890625) disk./.usagePercent‚83.17543485925589 disk.sdb2.in‚ 4.48828125 disk.sdb.out‚745.8623046875 disk.md0p1.out‚ 719.140625 disk.sdb2.out‚745.8623046875 disk.rbd0.out‚ 1.962890625 disk.sdb1.out‚0# disk./.usageGB‚2974.75643157959 disk.sda.out‚745.8623046875! disk.nvme0n1.out‚ 38177.109375 disk.dm-0.in‚ 530853.859375 disk.sda1.in‚0$ memory_percent‚36.08975410461426# proc.memory.rssMB‚ 3030.74609375 disk.sdb.in‚ 4.48828125 network.recv‚ 2.289358e+06 network.sent‚ 1.473283e+07 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 1.962890625yÔÐg2£- _wandbruntime‚6829  _runtime‚6829ªkªÇ¤Á _runtime‚6829.965213337 _step‚480 & lr-AdamW/pg1‚0.0005008528515923397 & lr-AdamW/pg2‚0.0005008528515923397 trainer/global_step‚5899 % _timestamp‚1.7884707767803307e+09L5‘ÈÌ¥Æ & lr-AdamW/pg2‚0.0005008528515923397 trainer/global_step‚5899 % _timestamp‚1.7884707767803307e+09 _runtime‚6829.965213337 _step‚480 & lr-AdamW/pg1‚0.0005008528515923397àåM{¸2¦- _wandbruntime‚6829  _runtime‚6829l¶ Ó §èO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚251090015356416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 160145060 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚239381919970304 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 160145060 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11707861717504 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 109090058 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66684384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 111711255 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11907872324096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚681 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884707767812939e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66684382 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 271829969 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42405676 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed¡l#‚ 160118716 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚239381901967360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 160118716 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚239381901967360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 271856315 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚251089763684864 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚251089763684864 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚683 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚239558412442670 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11674911973050 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11908156652032 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 111711253 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚227883729450704 _step‚481 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11707861717504 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 111711253 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚227883746490064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚239381919970304 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚265511556975616 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚253603684651520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11708095386112 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚251090015356416 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚265511865243648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚5899 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚239381919970304 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 160145060 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11707861717504 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 109090058 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66684384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 111711255 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11907872324096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚681 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884707767812939e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66684382 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 271829969 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42405676 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 160118716 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚239381901967360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 160118716 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚239381901967360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 271856315 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚251089763684864 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚251089763684864 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚683 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚239558412442670 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11674911973050 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11908156652032 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 111711253 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚227883729450704 _step‚481 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11707861717504 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 111711253 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚227883746490064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚239381919970304 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚265511556975616 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚253603684651520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11708095386112 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚251090015356416 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚265511865243648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚5899 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 271901308 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚239598147309732 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111729528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚239421188004352 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚227921259808128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11676870462244 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚251131265432576 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚239597901289080 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11676641480952 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11709825743360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11710059425280 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 271874936 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚251131265432576 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚239421206007296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11909877137408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚253645368621568 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11709825743360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111729528 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109108106 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66695186 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚6830.947517999 _step‚482 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 271874936 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 271901308 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 160145408 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚253645392561664 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0m!Á¥«÷M VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42413589 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚227921276847488 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 271901308 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚239598147309732 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111729528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚239421188004352 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚227921259808128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11676870462244 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚251131265432576 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚239597901289080 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11676641480952 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11709825743360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11710059425280 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 271874936 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚251131265432576 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚239421206007296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11909877137408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚253645368621568 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11709825743360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111729528 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109108106 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66695186 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚6830.947517999 _step‚482 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 271874936 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 271901308 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 160145408 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚253645392561664 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11710059425280 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚251131013747712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 111729530 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66695184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 160145408 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚5899 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0âŸüÛ2¬- _wandbruntime‚6830  _runtime‚6830\a¯^[­Õ 2loss/SRqqt2/assignment_loss‚1.5087436608970166 /loss/FRt3/detection_loss‚1.3984559679623036 1loss/SRqqt3/detection_loss‚0.8916932539591058 0loss/SRqqt4/detection_loss‚2.458277805098172 /loss/FBt2/detection_loss‚2.1775086283683778 _step‚483 1loss/FBt1/assignment_loss‚0.21885921484499704 .loss/FRt4/detection_loss‚3.086654211084048 /loss/FBt3/detection_loss‚0.5226018546684964 .loss/FBt4/detection_loss‚2.825091697952964 /loss/FRt2/detection_loss‚2.3320872255558656 1loss/SRqqt1/detection_loss‚1.8988391040609434 epoch‚4 _runtime‚6830.949447399 $ _timestamp‚1.788470777764736e+09 /loss/FRt1/assignment_loss‚2.655078790254063 1loss/SRqqt1/assignment_loss‚1.531583212602597 2loss/SRqqt4/assignment_loss‚1.3620286088051468 1loss/FBt4/assignment_loss‚0.03495810812842331 /loss/SRqqt2/detection_loss‚2.38227525750796 /loss/FBt1/detection_loss‚2.0970033407211304 0loss/FRt2/assignment_loss‚3.2868528095158664 0loss/FRt3/assignment_loss‚1.2619866229155485 2loss/FBt2/assignment_loss‚0.029206207961745653 trainer/global_step‚5899 2loss/SRqqt3/assignment_loss‚0.9291460493008014 %loss/total_loss‚41.08816189330683 /loss/FRt1/detection_loss‚2.7255048635933132 /loss/FRt4/assignment_loss‚3.438558466732502 1loss/FBt3/assignment_loss‚0.03516693081544382b—f`®Ú 1loss/SRqqt3/detection_loss‚0.8916932539591058 0loss/SRqqt4/detection_loss‚2.458277805098172 /loss/FBt2/detection_loss‚2.1775086283683778 _step‚483 1loss/FBt1/assignment_loss‚0.21885921484499704 .loss/FRt4/detection_loss‚3.086654211084048 /loss/FBt3/detection_loss‚0.5226018546684964 .loss/FBt4/detection_loss‚2.825091697952964 /loss/FRt2/detection_loss‚2.3320872255558656 1loss/SRqqt1/detection_loss‚1.8988391040609434 epoch‚4 _runtime‚6830.949447399 $ _timestamp‚1.788470777764736e+09 /loss/FRt1/assignment_loss‚2.655078790254063 1loss/SRqqt1/assignment_loss‚1.531583212602597 2loss/SRqqt4/assignment_loss‚1.3620286088051468 1loss/FBt4/assignment_loss‚0.03495810812842331 /loss/SRqqt2/detection_loss‚2.38227525750796 /loss/FBt1/detection_loss‚2.0970033407211304 0loss/FRt2/assignment_loss‚3.2868528095158664 0loss/FRt3/assignment_loss‚1.2619866229155485 2loss/FBt2/assignment_loss‚0.029206207961745653 trainer/global_step‚5899 2loss/SRqqt3/assignment_loss‚0.9291460493008014 %loss/total_loss‚41.08816189330683 /loss/FRt1/detection_loss‚2.7255048635933132 /loss/FRt4/assignment_loss‚3.438558466732502 1loss/FBt3/assignment_loss‚0.03516693081544382 2loss/SRqqt2/assignment_loss‚1.5087436608970166 /loss/FRt3/detection_loss‚1.3984559679623036ãs¶R’¯‚(:‡ ‚ÌçÔ϶õ‡ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚116.583' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.14575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ê'oƒ°‚(:ø ‚ÌçÔ…ùƈ disk.md0.in‚8.890625 disk.rbd2.in‚0# disk.nvme1n1.out‚48973.91015625) proc.memory.percent‚18.49820613861084 cpu‚25.160062641520202# disk.nvme0n1.out‚38177.31640625 disk.rbd0.in‚0.015625 disk.sdb.out‚747.7294921875 disk.sda.in‚ 4.40234375 disk.sda.out‚747.7294921875% memory_percent‚36.089372634887695 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sdb1.in‚0) proc.memory.availableMB‚ 10471.1171875) disk./.usagePercent‚83.17736189508973# disk.nvme0n1.in‚329776.41796875 disk.dm-0.in‚530856.1484375 disk.md0p1.out‚ 720.93359375 disk.sdb.in‚ 4.48828125 disk.dm-0.out‚ 87151.2265625 disk.sdb2.in‚ 4.48828125 disk.sdb2.out‚747.7294921875 disk.md0.out‚ 720.93359375 disk.md0p1.in‚8.890625 disk.rbd0.out‚ 1.962890625 disk.rbd2.out‚ 1.962890625 disk.sdb1.out‚0# proc.memory.rssMB‚ 3030.74609375 network.recv‚ 2.293607e+06$ disk./.usageGB‚2974.825351715088 disk.sda1.in‚0 disk.sda2.in‚ 4.40234375 disk.sda2.out‚747.7294921875# disk.nvme1n1.in‚201079.73046875 network.sent‚ 1.4751999e+07ß]‚±‚(:… ‘ÌçÔÑÊˇ gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚130.232' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.558 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@ôëd|²‚(:ñ ‘ÌçÔÌçÖ‡ network.recv‚ 2.300349e+06) disk./.usagePercent‚83.18085620078256 disk.dm-0.in‚530856.1484375 disk.sda1.in‚0 disk.md0.in‚8.890625 disk.md0.out‚ 722.40625 disk.sdb.out‚749.27294921875 disk.sdb1.out‚0 disk.sda.out‚749.27294921875 disk.dm-0.out‚ 87488.8125 disk.sdb2.in‚ 4.48828125" disk.nvme1n1.out‚ 49045.5703125* proc.memory.availableMB‚10471.10546875# proc.memory.rssMB‚ 3030.74609375) proc.memory.percent‚18.49820613861084$ disk./.usageGB‚2974.950325012207 proc.cpu.threads‚17 network.sent‚ 1.4846646e+07! disk.sda2.out‚749.27294921875! disk.sdb2.out‚749.27294921875 disk.sdb1.in‚0 disk.sda.in‚ 4.40234375# disk.nvme0n1.in‚329776.41796875 disk.rbd0.out‚ 1.962890625# disk.nvme1n1.in‚201079.73046875 disk.sdb.in‚ 4.48828125 disk.sda2.in‚ 4.40234375" disk.nvme0n1.out‚ 38443.2421875 disk.sda1.out‚0 disk.md0p1.in‚8.890625 disk.md0p1.out‚ 722.40625 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 1.962890625 disk.rbd2.in‚0% memory_percent‚36.089444160461426 cpu‚25.15979340285652À8¯‘³‚(:†  ÌçÔ¯êú… gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚135.802' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.9505 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gc´‚(:ö  ÌçÔ–É‹‡% disk./.usageGB‚2975.0756187438965" disk.nvme1n1.out‚ 49103.5078125 disk.sda.in‚ 4.40234375 disk.dm-0.in‚530856.1484375 disk.sda2.in‚ 4.40234375# disk.nvme0n1.in‚329776.41796875# disk.nvme0n1.out‚38643.06640625 disk.dm-0.out‚87746.57421875$ memory_percent‚36.09018325805664 disk.md0.in‚8.890625 disk.sdb.in‚ 4.48828125 disk.md0p1.out‚ 723.98046875( disk./.usagePercent‚83.1843594659601 disk.sdb2.out‚750.8994140625 disk.rbd0.out‚ 1.962890625 disk.sdb.out‚750.8994140625 disk.sda.out‚750.8994140625 network.sent‚ 1.4852596e+07 disk.sda2.out‚750.8994140625# proc.memory.rssMB‚ 3030.74609375 cpu‚25.159559309275767 disk.rbd2.out‚ 1.962890625 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.percent‚18.49820613861084 proc.cpu.threads‚17 disk.sdb2.in‚ 4.48828125 disk.md0.out‚ 723.98046875 disk.rbd2.in‚0# disk.nvme1n1.in‚201079.73046875 disk.sdb1.out‚0 disk.md0p1.in‚8.890625 network.recv‚ 2.304717e+06 disk.rbd0.in‚0.015625 disk.sdb1.in‚0( proc.memory.availableMB‚ 10470.984375ťܒµ‚(:‡ ¯ÌçÔľì… gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚140.145' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.03625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04xºjƒ¶‚(:ø ¯ÌçÔìÔ‹‡! disk.sda2.out‚752.93701171875 disk.rbd2.out‚ 1.962890625 network.recv‚ 2.308992e+06) disk./.usagePercent‚83.18434506678827 disk.sdb2.in‚ 4.48828125 disk.rbd2.in‚0 disk.sdb.out‚752.93701171875 disk.md0.in‚8.890625 disk.rbd0.out‚ 1.962890625" disk.nvme1n1.out‚ 49230.6328125 disk.sdb.in‚ 4.48828125 disk.sdb1.in‚0 disk.sda1.out‚0 disk.md0.out‚ 725.96484375 disk.sda.in‚ 4.40234375 disk.dm-0.out‚ 88006.71875# proc.memory.rssMB‚ 3030.74609375# disk.nvme0n1.in‚329776.41796875 disk.sda1.in‚0) proc.memory.percent‚18.49820613861084! disk.sdb2.out‚752.93701171875 disk.rbd0.in‚0.015625" disk.nvme0n1.out‚ 38776.0859375 disk.sda2.in‚ 4.40234375$ memory_percent‚36.09013557434082 cpu‚25.159321789767915 proc.cpu.threads‚17 network.sent‚ 1.4858271e+07 disk.md0p1.in‚8.890625 disk.md0p1.out‚ 725.96484375 disk.sdb1.out‚0 disk.dm-0.in‚530856.1484375) proc.memory.availableMB‚ 10470.9921875% disk./.usageGB‚2975.0751037597656# disk.nvme1n1.in‚201079.73046875 disk.sda.out‚752.93701171875º3ex2·- _wandbruntime‚6886  _runtime‚6886tœœƒÇ¸Á trainer/global_step‚5949 % _timestamp‚1.7884708332455769e+09 _runtime‚6886.430617879 _step‚484 & lr-AdamW/pg1‚0.0004950320234890209 & lr-AdamW/pg2‚0.0004950320234890209øEÌ¹Æ % _timestamp‚1.7884708332455769e+09 _runtime‚6886.430617879 _step‚484 & lr-AdamW/pg1‚0.0004950320234890209 & lr-AdamW/pg2‚0.0004950320234890209 trainer/global_step‚5949ä8Ç)2º- _wandbruntime‚6886  _runtime‚6886þä{›î'»èO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚682 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42751743 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚241532868354378 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚253152281241088 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 trainer/global_step‚5949 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚241533114374822 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 109976225 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚267695941232128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11806077424128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚253152532912640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275939328 % _timestamp‚1.7884708332467003e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚241346221819904 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112625005 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112625003 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚684 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299879424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚229760264361264 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11806077424128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 _step‚485 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚241346203816960 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112625003 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 109976909 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚255687883153920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11806311092736 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚229760247321904 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12008034138112 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚255687907094016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 161483012 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 161456668 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚253152281241088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚241346221819904 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _runtime‚6886.432168714 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67224482 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12007758198784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 274108017 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 161483012 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11806311092736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11772850013558 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚241346203816960 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42752425 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚267695641352704 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 161456668 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 274081671 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚241532868354378 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚253152281241088 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 trainer/global_step‚5949 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚241533114374822 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 109976225 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚267695941232128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11806077424128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚253152532912640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275939328 % _timestamp‚1.7884708332467003e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚241346221819904 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112625005 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112625003 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚684 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299879424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚229760264361264 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11806077424128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 _step‚485 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚241346203816960 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112625003 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 109976909 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚255687883153920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11806311092736 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚229760247321904 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12008034138112 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚255687907094016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 161483012 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 161456668 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚253152281241088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚241346221819904 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _runtime‚6886.432168714 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67224482 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12007758198784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ÔM8/ãG?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 274108017 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 161483012 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11806311092736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11772850013558 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚241346203816960 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42752425 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚267695641352704 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444å #2½- _wandbruntime‚6887  _runtime‚6887"îäø&¾òM ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11808275131904 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚267739318048256 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11774579521460 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112643278 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚241572603221440 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109994067 _runtime‚6887.485478799 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚241385489853952 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11808041449984 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚253193782988800 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11808275131904 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 274126638 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚241385507856896 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11808041449984 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67235284 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112643278 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 161509730 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚255729591064064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 161483360 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚659 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚253193782988800 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 274153010 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12009750924288 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112643280 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚241572357200788 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112643280 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67235286 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 274153010 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 161509730 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚255729567123968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12010041530368 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884708342992377e+09:;FÕý&¿÷M SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112643278 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚241572603221440 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109994067 _runtime‚6887.485478799 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚241385489853952 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11808041449984 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚253193782988800 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11808275131904 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 274126638 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚241385507856896 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11808041449984 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67235284 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112643278 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 161509730 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚255729591064064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 161483360 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚659 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚253193782988800 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 274153010 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12009750924288 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112643280 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚241572357200788 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112643280 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67235286 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 274153010 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 161509730 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚255729567123968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12010041530368 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884708342992377e+09 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11808275131904 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚267739318048256 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11774579521460 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360æ`ಠ2À- _wandbruntime‚6887  _runtime‚6887wnTZÁÔ 2loss/SRqqt1/assignment_loss‚1.4921393441872146 0loss/SRqqt2/detection_loss‚2.284339539932482 2loss/SRqqt4/assignment_loss‚1.5216797392702464 .loss/FRt1/detection_loss‚2.749059110879898 /loss/FRt2/detection_loss‚2.0662080259324282 /loss/FRt4/detection_loss‚3.0711070251464845 /loss/FBt2/detection_loss‚2.7759525179862976 /loss/FBt4/detection_loss‚2.4579942365487417 1loss/FBt4/assignment_loss‚0.15720935091361904 0loss/FRt3/assignment_loss‚1.4511520016577937 trainer/global_step‚5949 _step‚487 /loss/FRt1/assignment_loss‚2.552699575200677 _runtime‚6887.487364227 /loss/FRt2/assignment_loss‚3.763878379762173 0loss/FRt4/assignment_loss‚3.0225125694274904 2loss/SRqqt2/assignment_loss‚1.9202894257438001 1loss/SRqqt3/assignment_loss‚1.191509369592946 0loss/FBt1/assignment_loss‚0.3001598996023631 .loss/FRt3/detection_loss‚1.611178900025064 1loss/SRqqt3/detection_loss‚0.7466452399621138 % _timestamp‚1.7884708343026261e+09 1loss/FBt2/assignment_loss‚0.22474121689447202 1loss/SRqqt1/detection_loss‚1.9600596550183418 /loss/FBt1/detection_loss‚2.4200266549984613 .loss/FBt3/assignment_loss‚0.12222672443585 1loss/SRqqt4/detection_loss‚2.2481474099737224 epoch‚4 /loss/FBt3/detection_loss‚0.3871954882531153 &loss/total_loss‚42.498111401345795Éãíç_ÂÙ 2loss/SRqqt1/assignment_loss‚1.4921393441872146 0loss/SRqqt2/detection_loss‚2.284339539932482 2loss/SRqqt4/assignment_loss‚1.5216797392702464 .loss/FRt1/detection_loss‚2.749059110879898 /loss/FRt2/detection_loss‚2.0662080259324282 /loss/FRt4/detection_loss‚3.0711070251464845 /loss/FBt2/detection_loss‚2.7759525179862976 /loss/FBt4/detection_loss‚2.4579942365487417 1loss/FBt4/assignment_loss‚0.15720935091361904 0loss/FRt3/assignment_loss‚1.4511520016577937 trainer/global_step‚5949 _step‚487 /loss/FRt1/assignment_loss‚2.552699575200677 _runtime‚6887.487364227 /loss/FRt2/assignment_loss‚3.763878379762173 0loss/FRt4/assignment_loss‚3.0225125694274904 2loss/SRqqt2/assignment_loss‚1.9202894257438001 1loss/SRqqt3/assignment_loss‚1.191509369592946 0loss/FBt1/assignment_loss‚0.3001598996023631 .loss/FRt3/detection_loss‚1.611178900025064 1loss/SRqqt3/detection_loss‚0.7466452399621138 % _timestamp‚1.7884708343026261e+09 1loss/FBt2/assignment_loss‚0.22474121689447202 1loss/SRqqt1/detection_loss‚1.9600596550183418 /loss/FBt1/detection_loss‚2.4200266549984613 .loss/FBt3/assignment_loss‚0.12222672443585 1loss/SRqqt4/detection_loss‚2.2481474099737224 epoch‚4 /loss/FBt3/detection_loss‚0.3871954882531153 &loss/total_loss‚42.498111401345795ç,¸z‘Â(:† ¾ÌçԘÇ gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚138.642' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.6605 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚m©{Ä‚(:ð ¾ÌçÔÄŽçˆ disk.rbd0.in‚0.015625 disk.sda.out‚ 754.462890625 disk.dm-0.out‚ 88409.6953125 disk.md0p1.in‚8.890625 disk.sdb2.in‚ 4.48828125 disk.sdb2.out‚ 754.462890625 cpu‚25.158927978759007 network.sent‚ 1.4877414e+07 disk.sdb.out‚ 754.462890625 disk.sdb1.in‚0" disk.nvme0n1.out‚ 38975.6015625 disk.md0.in‚8.890625$ memory_percent‚36.09023094177246 disk.rbd0.out‚ 1.962890625 disk.rbd2.in‚0 disk.sda.in‚ 4.40234375 disk.sda2.out‚ 754.462890625) proc.memory.percent‚18.49820613861084 network.recv‚ 2.313507e+06# disk.nvme1n1.in‚201198.53515625" disk.nvme0n1.in‚329776.4296875 disk.md0.out‚ 727.42578125 disk.rbd2.out‚ 1.962890625 disk.sda1.in‚0% disk./.usageGB‚2974.8006439208984 disk.nvme1n1.out‚ 49434.09375 disk.sdb.in‚ 4.48828125 disk.sdb1.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚83.17667105482317 disk.dm-0.in‚530974.96484375 disk.sda1.out‚0 disk.md0p1.out‚ 727.42578125 disk.sda2.in‚ 4.40234375) proc.memory.availableMB‚ 10470.9765625# proc.memory.rssMB‚ 3030.74609375Lð‘Å‚(:† ÍÌçԠ轇 gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚136.466' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.1165 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ò¥‚Æ‚(:÷ ÍÌçÔÔ¼¨ˆ) proc.memory.percent‚18.49820613861084 cpu‚25.15855261304314 network.sent‚ 1.4974089e+07# disk.nvme1n1.out‚49593.89453125 disk.sdb1.out‚0 disk.md0p1.out‚ 728.66796875 proc.cpu.threads‚17 network.recv‚ 2.32076e+06 disk.rbd0.in‚0.015625! disk.sdb2.out‚755.76708984375 disk.md0p1.in‚8.890625# memory_percent‚36.0905647277832) disk./.usagePercent‚83.17889866003654 disk.rbd0.out‚ 1.962890625 disk.rbd2.out‚ 1.962890625 disk.sdb.in‚ 4.48828125 disk.sdb1.in‚0# disk.nvme0n1.in‚329776.44140625 disk.dm-0.out‚88768.94921875( proc.memory.availableMB‚ 10470.921875$ disk./.usageGB‚2974.880313873291 disk.sda.out‚755.76708984375 disk.sda1.in‚0 disk.sda2.in‚ 4.40234375 disk.sdb2.in‚ 4.48828125 disk.md0.in‚8.890625 disk.sdb.out‚755.76708984375 disk.dm-0.in‚530976.3828125 disk.md0.out‚ 728.66796875 disk.rbd2.in‚0 disk.sda.in‚ 4.40234375" disk.nvme0n1.out‚ 39175.0546875! disk.sda2.out‚755.76708984375# proc.memory.rssMB‚ 3030.74609375# disk.nvme1n1.in‚201199.94140625 disk.sda1.out‚05¾<ê’Ç‚(:‡ ÜÌçÔÒÙþ‡ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚113.181' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.29525 gpu.0.smClock‚1260 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£W—;~È‚(:ó ÜÌçÔŸØŠ‰) disk./.usagePercent‚83.17725864769463 disk.rbd2.out‚ 1.962890625 disk.sdb1.in‚0) proc.memory.percent‚18.49820613861084 disk.md0.in‚8.890625 disk.sdb.out‚757.3740234375 disk.sda.out‚757.3740234375 disk.sda2.out‚757.3740234375 network.recv‚ 2.324821e+06 disk.md0.out‚ 730.21484375 disk.rbd0.in‚0.015625# disk.nvme1n1.in‚201199.94140625 cpu‚25.158169494325755 disk.sda.in‚ 4.40234375# disk.nvme0n1.out‚39308.16796875 disk.dm-0.out‚ 88970.8984375 disk.sda1.out‚0 disk.md0p1.out‚ 730.21484375 disk.nvme0n1.in‚ 329776.46875 disk.dm-0.in‚530976.41015625 disk.sda1.in‚0$ memory_percent‚36.09123229980469 network.sent‚ 1.4977822e+07 disk.sdb2.in‚ 4.48828125 disk.sdb1.out‚0 disk.sda2.in‚ 4.40234375# proc.memory.rssMB‚ 3030.74609375# disk.nvme1n1.out‚49662.73046875% disk./.usageGB‚2974.8216590881348 disk.sdb2.out‚757.3740234375 disk.sdb.in‚ 4.48828125 disk.md0p1.in‚8.890625& proc.memory.availableMB‚ 10470.8125 disk.rbd0.out‚ 1.962890625 disk.rbd2.in‚0 proc.cpu.threads‚17+òù˜2É- _wandbruntime‚6944  _runtime‚6944l”ÇÊÁ & lr-AdamW/pg1‚0.0004891939388076823 & lr-AdamW/pg2‚0.0004891939388076823 trainer/global_step‚5999 % _timestamp‚1.7884708908305986e+09 _runtime‚6944.015532803 _step‚488 ¹-öÌËÆ % _timestamp‚1.7884708908305986e+09 _runtime‚6944.015532803 _step‚488 & lr-AdamW/pg1‚0.0004891939388076823 & lr-AdamW/pg2‚0.0004891939388076823 trainer/global_step‚5999è·kRG2Ì- _wandbruntime‚6944  _runtime‚6944Ž‚ g ÍèO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚243310505666560 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 162820964 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚257772105596416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚255215050468864 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113538755 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43098508 _step‚489 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚243310523669504 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 110862435 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11870788054066 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12107652890112 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚231636782232464 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11904293130752 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚257772081656320 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67764584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚255214798797312 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0¥ì¶ÿó'ÎíO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11870788054066 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12107652890112 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚231636782232464 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11904293130752 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚257772081656320 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67764584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚255214798797312 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚243310505666560 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 162820964 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚257772105596416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚255215050468864 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113538755 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43098508 _step‚489 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚243310523669504 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 110862435 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273éuD›©Ï‚(:… ëÌçÔø gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚128.056' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ ¼½yЂ(:î ëÌçÔëÙ݈ disk.dm-0.in‚530976.41015625 disk.dm-0.out‚89192.03515625 disk.sda1.out‚0# proc.memory.rssMB‚ 3030.74609375% disk./.usageGB‚2974.9509239196777 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.sda.out‚759.41064453125# disk.nvme1n1.out‚49715.76953125! disk.sdb2.out‚759.41064453125 network.recv‚ 2.32861e+06 disk.nvme0n1.in‚ 329776.46875 disk.sda2.in‚ 4.40234375! disk.sda2.out‚759.41064453125$ memory_percent‚36.09209060668945 proc.cpu.threads‚17( disk./.usagePercent‚83.1808729464861 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 1.962890625 disk.sdb1.in‚0 disk.sda1.in‚0 disk.sdb2.in‚ 4.48828125 disk.md0.in‚8.890625( proc.memory.availableMB‚ 10470.671875 network.sent‚ 1.4983162e+07 disk.rbd2.out‚ 1.962890625 disk.sdb.out‚759.41064453125 disk.sda.in‚ 4.40234375 disk.md0.out‚732.1875 cpu‚25.157760537679078# disk.nvme1n1.in‚201199.94140625 disk.sdb.in‚ 4.48828125! disk.nvme0n1.out‚ 39476.265625 disk.md0p1.in‚8.890625 disk.md0p1.out‚732.1875) proc.memory.percent‚18.49820613861084˜HÁ³2Ñ- _wandbruntime‚6945  _runtime‚69450’Тø&ÒòM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚5999 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 % _timestamp‚1.7884708918279474e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚257813765626368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113557030 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚679 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 276378340 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11906490838528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚255256300545024 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚269923747226624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚257813789566464 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12109957660160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276404712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 162821312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113557028 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚269923428486144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67775386 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚255256048860160 _runtime‚6945.013261645 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11906490838528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43105473 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11872746543260 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11906257156608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113557028 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 276404712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 162821312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43104794 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110880178 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 162847682 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚243349791703552 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11872517561968 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 _step‚490 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚243349809706496 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚231674295550528 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113557030 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 _step‚490 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚243349809706496 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚231674295550528 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113557030 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 276378340 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11906490838528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚255256300545024 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚269923747226624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚257813789566464 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12109957660160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276404712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 162821312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113557028 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚269923428486144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67775386 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚255256048860160 _runtime‚6945.013261645 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11906490838528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43105473 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11872746543260 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11906257156608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113557028 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 276404712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 162821312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43104794 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110880178 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 162847682 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚243349791703552êÑjl2Ô- _wandbruntime‚6945  _runtime‚6945”Fù³XÕÒ /loss/FRt1/assignment_loss‚3.667053782626202 /loss/FBt1/detection_loss‚3.1547445399420604 /loss/FBt2/detection_loss‚2.2707538306713104 /loss/FBt4/detection_loss‚3.0431943118572233 _step‚491 1loss/FBt1/assignment_loss‚0.12994099962153477 /loss/FRt1/detection_loss‚2.6042022901146034 _runtime‚ 6945.0143595 /loss/FRt2/assignment_loss‚3.341335342285481 2loss/SRqqt2/assignment_loss‚1.6861322414543893 2loss/SRqqt3/assignment_loss‚1.0256481921566385 0loss/SRqqt2/detection_loss‚2.444615074881801 0loss/SRqqt4/detection_loss‚2.671021843595164 2loss/SRqqt4/assignment_loss‚1.9658668291355883 0loss/FBt3/assignment_loss‚0.1920904820318506 /loss/FRt2/detection_loss‚2.0707903159039214 % _timestamp‚1.7884708918296728e+09 /loss/FRt3/detection_loss‚1.3533924455826098 0loss/FBt3/detection_loss‚0.46547057147364357 epoch‚4 trainer/global_step‚5999 /loss/FBt4/assignment_loss‚0.649511180807201 .loss/FRt4/detection_loss‚3.159566346804301 1loss/SRqqt1/detection_loss‚2.1515416357761774 &loss/total_loss‚44.960425515830025 /loss/FRt3/assignment_loss‚1.328964681808765 1loss/FBt2/assignment_loss‚0.29147231464072626 /loss/SRqqt3/detection_loss‚0.65997732345547 0loss/FRt4/assignment_loss‚3.1152180592219034 2loss/SRqqt1/assignment_loss‚1.5179208799814567|€§Õ]Ö× 0loss/FRt4/assignment_loss‚3.1152180592219034 2loss/SRqqt1/assignment_loss‚1.5179208799814567 /loss/FRt1/assignment_loss‚3.667053782626202 /loss/FBt1/detection_loss‚3.1547445399420604 /loss/FBt2/detection_loss‚2.2707538306713104 /loss/FBt4/detection_loss‚3.0431943118572233 _step‚491 1loss/FBt1/assignment_loss‚0.12994099962153477 /loss/FRt1/detection_loss‚2.6042022901146034 _runtime‚ 6945.0143595 /loss/FRt2/assignment_loss‚3.341335342285481 2loss/SRqqt2/assignment_loss‚1.6861322414543893 2loss/SRqqt3/assignment_loss‚1.0256481921566385 0loss/SRqqt2/detection_loss‚2.444615074881801 0loss/SRqqt4/detection_loss‚2.671021843595164 2loss/SRqqt4/assignment_loss‚1.9658668291355883 0loss/FBt3/assignment_loss‚0.1920904820318506 /loss/FRt2/detection_loss‚2.0707903159039214 % _timestamp‚1.7884708918296728e+09 /loss/FRt3/detection_loss‚1.3533924455826098 0loss/FBt3/detection_loss‚0.46547057147364357 epoch‚4 trainer/global_step‚5999 /loss/FBt4/assignment_loss‚0.649511180807201 .loss/FRt4/detection_loss‚3.159566346804301 1loss/SRqqt1/detection_loss‚2.1515416357761774 &loss/total_loss‚44.960425515830025 /loss/FRt3/assignment_loss‚1.328964681808765 1loss/FBt2/assignment_loss‚0.29147231464072626 /loss/SRqqt3/detection_loss‚0.65997732345547ëâ¸Ìׂ(:‚ úÌçÔ†³ßˆ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚101.1' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.275 gpu.0.smClock‚1245 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²f£Ç~Ø‚(:ó úÌçÔÛûð‰ network.sent‚ 1.5079498e+07 disk.sda.in‚ 4.40234375 disk.dm-0.in‚530976.41015625 disk.sdb2.out‚761.4150390625 disk.rbd0.out‚ 1.962890625 disk.rbd2.out‚ 1.962890625) proc.memory.percent‚18.49820613861084 disk.sda1.in‚0 disk.sda2.in‚ 4.40234375% memory_percent‚36.093878746032715" disk.nvme1n1.out‚ 49868.9609375 disk.sdb.in‚ 4.48828125 disk.sda2.out‚761.4150390625 disk.sdb2.in‚ 4.48828125* proc.memory.availableMB‚10470.37890625( disk./.usagePercent‚83.1843597859417 disk.sdb.out‚761.4150390625 disk.sdb1.out‚0 disk.md0p1.in‚8.890625 disk.md0.in‚8.890625 disk.nvme0n1.in‚ 329776.46875! disk.nvme0n1.out‚ 39707.578125 disk.sda1.out‚0 disk.sdb1.in‚0 disk.md0.out‚ 734.12890625 disk.rbd0.in‚0.015625% disk./.usageGB‚2975.0756301879883# disk.nvme1n1.in‚201199.94140625 disk.sda.out‚761.4150390625 disk.md0p1.out‚ 734.12890625 cpu‚25.15741352569104 proc.cpu.threads‚17 network.recv‚ 2.335373e+06 disk.dm-0.out‚ 89576.5390625 disk.rbd2.in‚0# proc.memory.rssMB‚ 3030.74609375²}‘{Ù‚(:ð ‰ÍçÔ—¬ˆ% memory_percent‚36.092352867126465 disk.dm-0.out‚ 89811.2890625# disk.nvme1n1.out‚49970.69140625 proc.cpu.threads‚17 disk.md0p1.out‚735.4375 disk.sdb2.out‚762.7802734375 disk.md0.out‚735.4375# proc.memory.rssMB‚ 3030.74609375 cpu‚25.157052314999643 disk.sda2.in‚ 4.40234375 disk.md0p1.in‚8.890625 disk.md0.in‚8.890625 disk.sda.in‚ 4.40234375 disk.sda.out‚762.7802734375) disk./.usagePercent‚83.18436138584968% disk./.usageGB‚2975.0756874084473 disk.rbd0.in‚0.015625 disk.sdb.in‚ 4.48828125 disk.sdb.out‚762.7802734375 network.recv‚ 2.341829e+06 disk.dm-0.in‚530976.41015625 disk.sda1.in‚0 disk.sdb2.in‚ 4.48828125 disk.rbd2.in‚0 network.sent‚ 1.5178589e+07 disk.rbd0.out‚ 1.962890625 disk.rbd2.out‚ 1.962890625# disk.nvme1n1.in‚201199.94140625* proc.memory.availableMB‚10470.62890625) proc.memory.percent‚18.49820613861084# disk.nvme0n1.out‚39840.59765625 disk.nvme0n1.in‚ 329776.46875 disk.sda1.out‚0 disk.sda2.out‚762.7802734375 disk.sdb1.in‚0 disk.sdb1.out‚0ݼFì‘Ú‚(:† ‰ÍçÔЃð‡ gpu.0.gpu‚28.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚138.302' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.5755 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°ïæ‘Û‚(:† ˜ÍçÔšŽŠ† gpu.0.gpu‚37.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚113.466' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.3665 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0d¿iï|Ü‚(:ñ ˜ÍçԦᇈ disk.sdb.out‚764.56494140625 disk.sda.out‚764.56494140625 disk.dm-0.in‚530976.41015625 disk.sda1.in‚0 network.recv‚ 2.345962e+06) disk./.usagePercent‚83.17667894770256 proc.cpu.threads‚17 disk.sdb2.in‚ 4.48828125 disk.md0.in‚8.890625 cpu‚25.15671911887389 disk.sdb1.in‚0 disk.rbd0.in‚0.015625# disk.nvme1n1.out‚50022.92578125 disk.nvme0n1.in‚ 329776.46875 disk.sda2.in‚ 4.40234375! disk.sda2.out‚764.56494140625 disk.md0p1.out‚ 737.15625) proc.memory.availableMB‚ 10470.4296875 disk.rbd2.out‚ 1.962890625 disk.sdb1.out‚0 disk.sda.in‚ 4.40234375$ memory_percent‚36.09356880187988# proc.memory.rssMB‚ 3030.74609375) proc.memory.percent‚18.49820613861084 network.sent‚ 1.5184644e+07 disk.md0.out‚ 737.15625 disk.rbd2.in‚0! disk.nvme0n1.out‚ 39929.359375 disk.dm-0.out‚89942.03515625 disk.sdb.in‚ 4.48828125 disk.sda1.out‚0 disk.md0p1.in‚8.890625$ disk./.usageGB‚2974.800926208496! disk.sdb2.out‚764.56494140625 disk.rbd0.out‚ 1.962890625# disk.nvme1n1.in‚201199.94140625ŽAxlzÝ‚(:ï §ÍçÔ‘ôï† network.sent‚ 1.5187698e+07 disk.md0.out‚738.5625 disk.nvme1n1.out‚ 50066.4375 disk.sda.in‚ 4.40234375 disk.sdb.out‚766.03662109375 disk.sda.out‚766.03662109375 disk.dm-0.in‚530976.41015625 disk.sda2.in‚ 4.40234375 cpu‚25.162112568589613 proc.cpu.threads‚17 disk.md0.in‚8.890625 disk.sda1.out‚0) disk./.usagePercent‚83.17544179219047 disk.rbd2.out‚ 1.962890625 disk.sdb.in‚ 4.48828125$ disk./.usageGB‚2974.756679534912! disk.sdb2.out‚766.03662109375 disk.sdb1.out‚0 disk.md0p1.in‚8.890625* proc.memory.availableMB‚10470.43359375# proc.memory.rssMB‚ 3030.74609375 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 1.962890625 disk.rbd2.in‚0# disk.nvme1n1.in‚201199.94140625# disk.nvme0n1.out‚40106.72265625 disk.dm-0.out‚90173.16015625 disk.md0p1.out‚738.5625$ memory_percent‚36.09354496002197 disk.sdb2.in‚ 4.48828125 disk.nvme0n1.in‚ 329776.46875 disk.sda1.in‚0! disk.sda2.out‚766.03662109375 network.recv‚ 2.349879e+06 disk.sdb1.in‚0) proc.memory.percent‚18.49820613861084ý½ê’Þ‚(:‡ §ÍçÔÀÝχ gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0UwÝ‘ß‚(:† ¶ÍçÔ÷Êæ… gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}}à‚(:ò ¶ÍçÔ°“‡‡ disk.sdb2.in‚ 4.48828125 disk.sda.in‚ 4.40234375 disk.dm-0.in‚530976.41015625# disk.nvme1n1.out‚50066.64453125 network.sent‚ 1.5193987e+07% disk./.usageGB‚2974.8236198425293 disk.sdb1.in‚0 disk.dm-0.out‚90441.52734375 disk.md0p1.in‚8.890625 cpu‚25.17055659181947 network.recv‚ 2.354467e+06 disk.md0.in‚8.890625 disk.md0.out‚ 740.234375 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 1.962890625 disk.rbd2.in‚0* proc.memory.availableMB‚10469.69921875) proc.memory.percent‚18.49820613861084 disk.sdb.out‚767.7685546875 disk.nvme0n1.in‚ 329776.46875 disk.sda2.out‚767.7685546875# disk.nvme1n1.in‚201199.94140625 disk.rbd2.out‚ 1.962890625 disk.sdb.in‚ 4.48828125" disk.nvme0n1.out‚ 40374.8828125 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda2.in‚ 4.40234375$ memory_percent‚36.09802722930908 disk.sda.out‚767.7685546875 disk.sdb2.out‚767.7685546875 disk.sdb1.out‚0 disk.md0p1.out‚ 740.234375# proc.memory.rssMB‚ 3030.74609375) disk./.usagePercent‚83.17731347120814Ó͸G’á‚(:‡ ÅÍçÔÛÛ³ˆ gpu.0.gpu‚47.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚143.263' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.81575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†Å©â‚(:ö ÅÍçԎЗŠ disk.sda2.in‚ 4.40234375 disk.md0p1.out‚ 741.84765625$ memory_percent‚36.08591556549072 network.sent‚ 1.5199458e+07 network.recv‚ 2.358102e+06) disk./.usagePercent‚83.18087401309143 disk.sdb.out‚769.44775390625 disk.sdb1.in‚0 disk.sda.out‚769.44775390625! disk.sdb2.out‚769.44775390625 disk.md0p1.in‚8.890625 disk.md0.in‚8.890625 disk.rbd2.in‚0 disk.sdb.in‚ 4.48828125 disk.sda.in‚ 4.40234375 disk.nvme0n1.in‚ 329776.46875 disk.nvme0n1.out‚ 40627.3125 disk.dm-0.in‚530976.41015625 disk.rbd0.in‚0.015625 disk.sda1.out‚0 cpu‚25.179438171044655 proc.cpu.threads‚17% disk./.usageGB‚2974.9509620666504 disk.rbd0.out‚ 1.962890625) proc.memory.percent‚18.49820613861084 disk.md0.out‚ 741.84765625" disk.nvme1n1.out‚ 50066.7265625 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sdb2.in‚ 4.48828125* proc.memory.availableMB‚10471.68359375! disk.sda2.out‚769.44775390625 disk.rbd2.out‚ 1.962890625# proc.memory.rssMB‚ 3030.74609375# disk.nvme1n1.in‚201199.94140625 disk.dm-0.out‚ 90694.0390625½0n"ã‚(:… ÔÍçÔ ¦Ã‡ gpu.0.gpu‚36.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚142.93' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.7325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;‰«=~ä‚(:ó ÔÍçÔÛ늈) disk./.usagePercent‚83.18435999926275 disk.sda.in‚ 4.40234375 disk.sda.out‚771.14501953125 network.sent‚ 1.5207268e+07 disk.md0p1.in‚8.890625* proc.memory.availableMB‚10471.67578125# disk.nvme1n1.in‚201199.94140625" disk.nvme0n1.out‚ 40780.0234375 disk.sda1.in‚0# proc.memory.rssMB‚ 3030.74609375 disk.sdb2.in‚ 4.48828125 disk.rbd0.out‚ 1.962890625 disk.sdb.in‚ 4.48828125 disk.nvme0n1.in‚ 329776.46875 disk.sda1.out‚0 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.sdb1.in‚0 disk.sda2.in‚ 4.40234375$ disk./.usageGB‚2975.075637817383! disk.nvme1n1.out‚ 50067.078125 disk.sdb1.out‚0$ memory_percent‚36.08596324920654 cpu‚25.187861761677574! disk.sdb2.out‚771.14501953125 disk.md0.in‚8.890625 disk.md0.out‚ 743.484375 disk.rbd2.out‚ 1.962890625 disk.dm-0.out‚ 90847.1015625 disk.md0p1.out‚ 743.484375 proc.cpu.threads‚17 network.recv‚ 2.362343e+06 disk.sdb.out‚771.14501953125 disk.dm-0.in‚530976.41015625! disk.sda2.out‚771.14501953125) proc.memory.percent‚18.49820613861084 ÑW‡2å- _wandbruntime‚7054  _runtime‚7054meº T*æÎT K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 279599377 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚258608821712384 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚12222932944384 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚12018116118528 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 347418624 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚234615601505936 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚234615584466576 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32006 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚390 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚68833894 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20713 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3532 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1300 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 560288777 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 279620090 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 trainer/global_step‚6044 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 164466256 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚258609015527424 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚246590899408896 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚43708048 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 572522496 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 175812096 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 115133123 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193815040 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 193815040 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6236507136 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚309 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 115133121 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚258609015527424 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 112541942 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6458321920 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 164466256 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚309 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚246590881405952 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 _runtime‚7054.720658792 _step‚492 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚12223090579456 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 164486967 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 112541552 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚388 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚582 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚246599644032174 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6741059457 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚258608821712384 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚246590881405952 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚309 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚273 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188859832 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6966900736 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1319 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚43707660 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 157635072 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚11984042526238ËÄó™ÆçÓT BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 trainer/global_step‚6044 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 164466256 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚258609015527424 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚246590899408896 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated\—l‰“"‚43708048 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 572522496 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 175812096 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 115133123 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193815040 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 193815040 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6236507136 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚309 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 115133121 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚258609015527424 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 112541942 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6458321920 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 164466256 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚309 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚246590881405952 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 _runtime‚7054.720658792 _step‚492 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚12223090579456 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 164486967 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 112541552 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚388 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚582 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚246599644032174 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6741059457 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚258608821712384 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚246590881405952 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚309 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚273 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188859832 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6966900736 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1319 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚43707660 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 157635072 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚11984042526238 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 279599377 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚258608821712384 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚12222932944384 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚12018116118528 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 347418624 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚234615601505936 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚234615584466576 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32006 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚390 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚68833894 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20713 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3532 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1300 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 560288777 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 279620090 FDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚273431220338176 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚12223564439552 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚12223406804480 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3532 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚246620965083136 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚258639555061760 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 _step‚493[KÔÍd)êÞR \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚246625675889618 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 6236507136 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 572522496 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6966900736 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20713 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚36 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚246625487029786 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 565025792 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 193815040 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188859832 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 115148503 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚246620965083136 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 565025792 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚261207655898624 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚309 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚309 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 572522496 % _timestamp‚1.7884710026132102e+09 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 175812096 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚258639361246720 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3532 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚11984514734050 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6741059457 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚273431220338176 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚12223564439552 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚12223406804480 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3532 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚246620965083136 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚258639555061760 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 _step‚493 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 164475164 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 279644380 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚390 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚273431038763008 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 279623667 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20711 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚582 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚12018414166528 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚246620947080192 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 164495875 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚12018589978624 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 115148505 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚68845574 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚388 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 175812096 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚273 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32006 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚12018589978624 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 279623667 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 7134511104 trainer/global_step‚6044 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚258639361246720 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚261207631958528 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚234641144116208 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚582 _runtime‚7055.800272357 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 115148505 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 164495875 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 112559522 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 112559132 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 347418624 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6561988608 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 115148503 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚68845576 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 181575168 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚582 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 164475164í*¥V(’ë‚(:‡ ãÍçÔÑ̈ gpu.0.gpu‚42.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.357' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.83925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 [:¸wì‚(:ì ãÍçÔƒÙêˆ network.sent‚ 1.52232e+07 disk.sdb.in‚ 4.48828125 disk.sda.in‚ 4.40234375 disk.md0p1.in‚8.890625 disk.md0.in‚8.890625 disk.rbd0.in‚0.015625) proc.memory.percent‚18.49820613861084 disk.sdb.out‚ 772.474609375 disk.rbd0.out‚ 1.962890625 proc.cpu.threads‚17 disk.md0.out‚ 744.7578125" disk.nvme0n1.out‚ 41134.0078125 disk.sda2.in‚ 4.40234375 disk.md0p1.out‚ 744.7578125$ memory_percent‚36.08746528625488# proc.memory.rssMB‚ 3030.74609375) disk./.usagePercent‚83.18435999926275# disk.nvme1n1.out‚50067.19140625 disk.sda.out‚ 772.474609375 disk.sda1.out‚0 disk.sdb2.out‚ 772.474609375 disk.rbd2.out‚ 1.962890625# disk.nvme1n1.in‚201199.94140625 disk.sdb1.in‚0 disk.dm-0.out‚91201.19921875 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.sdb2.in‚ 4.48828125 network.recv‚ 2.366246e+06$ disk./.usageGB‚2975.075637817383 disk.nvme0n1.in‚ 329776.46875 disk.dm-0.in‚530976.41015625 disk.sda2.out‚ 772.474609375 disk.sda1.in‚0) proc.memory.availableMB‚ 10471.4296875 cpu‚25.19703902783776Qû¢«2í- _wandbruntime‚7071  _runtime‚70713v¬î¥§ 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9078014492988586 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.07381414622068405 particle/accuracy_1_of_5‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5236550569534302 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5791667103767395 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4479166269302368 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2299382984638214 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.49815621972084045 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9784754514694214 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47681936621665955 9Æ0Ûù"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5907362699508667 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8220093250274658 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04447571560740471 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_5‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6160256862640381 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 jet/accuracy_5_of_6‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6666666865348816 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.017989417538046837 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9987789988517761 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6388888955116272 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_2‚0.11629155278205872 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.22727274894714355 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22182969748973846 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9917044043540955 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.38345465064048767 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3590277433395386 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0833333358168602 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5524604320526123 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 jet/accuracy_4_of_7‚0 particle/accuracy_7_of_7‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5309125781059265 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5021488070487976 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 )loss/val_total_loss‚42.12470092394498 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.535739004611969 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6176470518112183 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 particle/accuracy_4_of_5‚1 particle/accuracy_5_of_5‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5207196474075317 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9296309351921082 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1666666865348816 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.37835100293159485 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.09783665835857391 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.019607843831181526 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5180104374885559 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.43560609221458435 9"Purity/*FRt0SRqqt4FBt/event_purity‚0.4000000059604645 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ;"Purity/2FRt1SRqqt*FBt/event_purity‚0.002897455357015133 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 :"Purity/1FRt2SRqqt0FBt/event_purity‚0.00266400258988142 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.4000000059604645 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7127236723899841 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2746184468269348 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5259209275245667 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0833333358168602 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_7‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9687500596046448 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.4166666567325592 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.39130446314811707 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.16382382810115814 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7272727489471436 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.3333333432674408 jet/accuracy_3_of_7‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9887398481369019 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5207196474075317 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.23423424363136292 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5432093143463135 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_7‚1 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05833333358168602 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15236026048660278 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9969838261604309 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9992063641548157 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.025902055203914642 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 particle/accuracy_5_of_6‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.26947924494743347 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5317391157150269 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5107136368751526 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4990590512752533 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.47341951727867126 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_2_of_6‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9917800426483154 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5509259700775146 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9833333492279053 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18715353310108185 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4787662625312805 ("Purity/1FRt0SRqqt4FBt/event_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3541666567325592 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9897370338439941 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9868040084838867 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4634501338005066 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5346187949180603 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7596009373664856 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4573643207550049 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.015949174761772156 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1184210553765297 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991621971130371 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9884612560272217 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5336190462112427 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.022967476397752762 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5120187401771545 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07645875960588455 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.11001060158014297 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9788978099822998 8"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.436015248298645 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12466300278902054 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_2_of_7‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06812994182109833 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_6‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 % _timestamp‚1.7884710178617623e+09 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0076230415143072605 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9969069361686707 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4835059940814972 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.418902724981308 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 jet/accuracy_1_of_6‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1554090976715088 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.023255813866853714 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5597010254859924 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 -jet/accuracy_5_of_5‚0.0026881720405071974 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13475178182125092 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9882716536521912 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7validation_average_jet_accuracy‚0.44165197014808655 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4934210479259491 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1498519629240036 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.500102162361145 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5100627541542053 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1626984179019928 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4374300241470337 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9992063641548157 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5864785313606262 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8874268531799316 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.629960298538208 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0015873016091063619 *jet/accuracy_2_of_5‚0.5144649147987366 particle/accuracy_1_of_3‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.012047438882291317 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2271825224161148 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5130900740623474 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735632538795471 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5553194284439087 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5833333730697632 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9900000095367432 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.34151050448417664 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.14965024590492249 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.625 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3958333432674408 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 *particle/specificity‚0.950347363948822 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 *"Purity/1FRt*SRqqt4FBt/event_purity‚0.5 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0173611119389534 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9798850417137146 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 +particle/sensitivity‚0.3384208083152771 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_6_of_6‚1 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.42119815945625305 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03289217874407768 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9897714257240295 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.006896551698446274 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4181681275367737 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5351852178573608 *jet/accuracy_1_of_5‚0.9541154503822327 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20067960023880005 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7424241900444031 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 )jet/accuracy_2_of_6‚0.692307710647583 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.45359480381011963 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2546902596950531 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04022988677024841 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1554090976715088 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21365214884281158 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5732378959655762 jet/accuracy_6_of_7‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2007545828819275 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5333333015441895 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4205513596534729 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14273376762866974 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4281676113605499 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.2625860869884491 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9712302088737488 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4995322823524475 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.18989771604537964 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_7_of_7‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9844487905502319 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2259463369846344 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9883788228034973 +jet/accuracy_2_of_4‚0.39058929681777954 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3465709090232849 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1350574791431427 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.35092589259147644 particle/accuracy_5_of_7‚1 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4208753705024719 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4000000059604645 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10151781141757965 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.019607843831181526 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.31481480598449707 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9977836608886719 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14738236367702484 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.004138834774494171 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 *jet/accuracy_1_of_3‚0.8752602338790894 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.24166664481163025 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1489114761352539 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.14645400643348694 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5289749503135681 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.016850775107741356 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5793934464454651 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.027764977887272835 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_4‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.17307692766189575 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9887377619743347 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2044992297887802 particle/accuracy_3_of_7‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.32055217027664185 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6382645964622498 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.46176475286483765 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.464702844619751 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.11625514179468155 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.43839186429977417 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5495588779449463 +jet/accuracy_3_of_3‚0.02775885909795761 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.14068584144115448 'particle/f_score‚0.4210241734981537 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3827160596847534 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.15625 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19223587214946747 particle/accuracy_2_of_4‚1 (particle/accuracy‚0.8547919392585754 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9952857494354248 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.2048906683921814 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.990070641040802 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.36265289783477783 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.51861971616745 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.24913790822029114 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5648148059844971 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4998551607131958 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5039193034172058 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151560306549 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5724515318870544 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.15625 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.27454420924186707 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1875 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9884465336799622 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.13821838796138763 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.028032345697283745 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.032294027507305145 particle/accuracy_2_of_7‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16232192516326904 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15983940660953522 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 <"Purity/2FRt1SRqqt1FBt/event_purity‚0.0059523810632526875 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07217181473970413 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2610529065132141 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.011194624938070774 6 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.897569477558136 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09927898645401001 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.997982919216156 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22260026633739471 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 particle/accuracy_1_of_6‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 _runtime‚7071.049818566 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6315789222717285 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6666666865348816 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9898540377616882 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6326388716697693 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3541666567325592 +jet/accuracy_4_of_6‚0.01923076994717121 jet/accuracy_1_of_7‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1601296365261078 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 particle/accuracy_3_of_4‚1 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1417970210313797 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5173439979553223 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5522488951683044 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.39747026562690735 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3856487572193146 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.13859649002552032 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5631081461906433 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7857142686843872 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0èÃN¿®.6666666865348816 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2684353291988373 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.030609600245952606 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1424148678779602 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4466666877269745 particle/accuracy_1_of_7‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9892356991767883 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5104166865348816 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4515969455242157 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1875 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9852794408798218 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2916666567325592 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5725358724594116 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1560344249010086 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.7105263471603394 trainer/global_step‚6044 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 *jet/accuracy_1_of_1‚0.4875940680503845 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 epoch‚4 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9989418387413025 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_6_of_7‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13249579071998596 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9100529551506042 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2202652245759964 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16163134574890137 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 _step‚494 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5581700801849365 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 particle/accuracy_3_of_3‚1 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 jet/accuracy_6_of_6‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5867063403129578 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2665676772594452 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.18470676243305206 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.292397677898407 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1490611881017685 ,jet/accuracy_4_of_5‚0.004480286967009306 particle/accuracy_1_of_1‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.47152358293533325 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.38109642267227173ƒM=Dp盛 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.22727274894714355 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22182969748973846 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9917044043540955 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.38345465064048767 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3590277433395386 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0833333358168602 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5524604320526123 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 jet/accuracy_4_of_7‚0 particle/accuracy_7_of_7‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5309125781059265 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5021488070487976 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 )loss/val_total_loss‚42.12470092394498 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.535739004611969 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6176470518112183 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 particle/accuracy_4_of_5‚1 particle/accuracy_5_of_5‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5207196474075317 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9296309351921082 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1666666865348816 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.37835100293159485 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.09783665835857391 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.019607843831181526 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5180104374885559 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.43560609221458435 9"Purity/*FRt0SRqqt4FBt/event_purity‚0.4000000059604645 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ;"Purity/2FRt1SRqqt*FBt/event_purity‚0.002897455357015133 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 :"Purity/1FRt2SRqqt0FBt/event_purity‚0.00266400258988142 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.4000000059604645 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7127236723899841 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2746184468269348 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5259209275245667 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0833333358168602 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_7‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9687500596046448 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.4166666567325592 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.39130446314811707 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.16382382810115814 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7272727489471436 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.3333333432674408 jet/accuracy_3_of_7‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9887398481369019 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5207196474075317 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.23423424363136292 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5432093143463135 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_7‚1 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05833333358168602 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15236026048660278 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9969838261604309 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9992063641548157 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.025902055203914642 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 particle/accuracy_5_of_6‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.26947924494743347 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5317391157150269 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5107136368751526 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4990590512752533 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.47341951727867126 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_2_of_6‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9917800426483154 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5509259700775146 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9833333492279053 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18715353310108185 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4787662625312805 ("Purity/1FRt0SRqqt4FBt/event_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3541666567325592 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9897370338439941 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9868040084838867 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4634501338005066 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5346187949180603 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7596009373664856 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4573643207550049 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.015949174761772156 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1184210553765297 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991621971130371 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9884612560272217 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5336190462112427 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.022967476397752762 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5120187401771545 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07645875960588455 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.11001060158014297 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9788978099822998 8"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.436015248298645 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12466300278902054 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_2_of_7‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06812994182109833 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_6‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 % _timestamp‚1.7884710178617623e+09 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0076230415143072605 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9969069361686707 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4835059940814972 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.418902724981308 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 jet/accuracy_1_of_6‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1554090976715088 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.023255813866853714 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5597010254859924 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 -jet/accuracy_5_of_5‚0.0026881720405071974 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13475178182125092 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9882716536521912 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7validation_average_jet_accuracy‚0.44165197014808655 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4934210479259491 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1498519629240036 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.500102162361145 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5100627541542053 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1626984179019928 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4374300241470337 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9992063641548157 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5864785313606262 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8874268531799316 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.629960298538208 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0015873016091063619 *jet/accuracy_2_of_5‚0.5144649147987366 particle/accuracy_1_of_3‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.012047438882291317 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2271825224161148 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5130900740623474 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735632538795471 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5553194284439087 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5833333730697632 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9900000095367432 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.34151050448417664 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.14965024590492249 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.625 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3958333432674408 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 *particle/specificity‚0.950347363948822 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 *"Purity/1FRt*SRqqt4FBt/event_purity‚0.5 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0173611119389534 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9798850417137146 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 +particle/sensitivity‚0.3384208083152771 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_6_of_6‚1 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.42119815945625305 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03289217874407768 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9897714257240295 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.006896551698446274 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4181681275367737 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5351852178573608 *jet/accuracy_1_of_5‚0.9541154503822327 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20067960023880005 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7424241900444031 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 )jet/accuracy_2_of_6‚0.692307710647583 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.45359480381011963 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2546902596950531 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04022988677024841 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1554090976715088 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21365214884281158 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5732378959655762 jet/accuracy_6_of_7‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2007545828819275 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5333333015441895 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4205513596534729 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14273376762866974 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4281676113605499 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.2625860869884491 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9712302088737488 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4995322823524475 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.18989771604537964 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_7_of_7‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9844487905502319 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2259463369846344 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9883788228034973 +jet/accuracy_2_of_4‚0.39058929681777954 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3465709090232849 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1350574791431427 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.35092589259147644 particle/accuracy_5_of_7‚1 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4208753705024719 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4000000059604645 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10151781141757965 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.019607843831181526 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.31481480598449707 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9977836608886719 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14738236367702484 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.004138834774494171 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 *jet/accuracy_1_of_3‚0.8752602338790894 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.24166664481163025 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1489114761352539 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.14645400643348694 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5289749503135681 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.016850775107741356 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5793934464454651 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.027764977887272835 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_4‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.17307692766189575 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9887377619743347 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2044992297887802 particle/accuracy_3_of_7‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.32055217027664185 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6382645964622498 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.46176475286483765 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.464702844619751 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.11625514179468155 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.43839186429977417 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5495588779449463 +jet/accuracy_3_of_3‚0.02775885909795761 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.14068584144115448 'particle/f_score‚0.4210241734981537 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3827160596847534 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.15625 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19223587214946747 particle/accuracy_2_of_4‚1 (particle/accuracy‚0.8547919392585754 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9952857494354248 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.2048906683921814 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.990070641040802 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.36265289783477783 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.51861971616745 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.24913790822029114 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5648148059844971 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4998551607131958 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5039193034172058 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151560306549 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5724515318870544 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.15625 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.27454420924186707 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1875 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9884465336799622 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.13821838796138763 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.028032345697283745 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.032294027507305145 particle/accuracy_2_of_7‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16232192516326904 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15983940660953522 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 <"Purity/2FRt1SRqqt1FBt/event_purity‚0.0059523810632526875 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07217181473970413 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2610529065132141 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.011194624938070774 6 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.897569477558136 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09927898645401001 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.997982919216156 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22260026633739471 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 particle/accuracy_1_of_6‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 _runtime‚7071.049818566 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6315789222717285 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 9"Purity/Cãöm#3FRt3SRqqt*FBt/SRqqt_purity‚0.6666666865348816 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9898540377616882 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6326388716697693 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3541666567325592 +jet/accuracy_4_of_6‚0.01923076994717121 jet/accuracy_1_of_7‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1601296365261078 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 particle/accuracy_3_of_4‚1 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1417970210313797 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5173439979553223 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5522488951683044 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.39747026562690735 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3856487572193146 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.13859649002552032 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5631081461906433 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7857142686843872 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2684353291988373 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.030609600245952606 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1424148678779602 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4466666877269745 particle/accuracy_1_of_7‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9892356991767883 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5104166865348816 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4515969455242157 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1875 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9852794408798218 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2916666567325592 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5725358724594116 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1560344249010086 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.7105263471603394 trainer/global_step‚6044 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 *jet/accuracy_1_of_1‚0.4875940680503845 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 epoch‚4 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9989418387413025 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_6_of_7‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13249579071998596 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9100529551506042 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2202652245759964 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16163134574890137 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 _step‚494 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5581700801849365 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 particle/accuracy_3_of_3‚1 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 jet/accuracy_6_of_6‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5867063403129578 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2665676772594452 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.18470676243305206 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.292397677898407 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1490611881017685 ,jet/accuracy_4_of_5‚0.004480286967009306 particle/accuracy_1_of_1‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.47152358293533325 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.38109642267227173 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9078014492988586 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.07381414622068405 particle/accuracy_1_of_5‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5236550569534302 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5791667103767395 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4479166269302368 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2299382984638214 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.49815621972084045 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9784754514694214 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47681936621665955 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5907362699508667 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8220093250274658 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04447571560740471 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_5‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6160256862640381 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 jet/accuracy_5_of_6‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6666666865348816 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.017989417538046837 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9987789988517761 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6388888955116272 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_2‚0.11629155278205872 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05î.QÞ2ð- _wandbruntime‚7079  _runtime‚7079þ@eÇñÁ _runtime‚7079.620058913 _step‚495 & lr-AdamW/pg1‚0.0004833398141964215 & lr-AdamW/pg2‚0.0004833398141964215 trainer/global_step‚6049 % _timestamp‚1.7884710264351697e+09 @·ÌòÆ & lr-AdamW/pg1‚0.0004833398141964215 & lr-AdamW/pg2‚0.0004833398141964215 trainer/global_step‚6049 % _timestamp‚1.7884710264351697e+09 _runtime‚7079.620058913 _step‚495ïZúÈ”2ó- _wandbruntime‚7079  _runtime‚7079‹¢4í'ôçO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 164695713 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12237502559232 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11998263313024 _runtime‚7079.621156829 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 280117282 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115421569 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚261769070635520 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚247168962996224 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 280143628 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚247168962996224 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12032379014144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69040648 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43817937 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12032379014144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69040650 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 164722057 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚259201575678976 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 112858587 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115421569 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚631 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚629 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚247168944993280 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 _step‚496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 164722057 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12237788984320 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259201324007424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚259201575678976 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115421571 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43817308 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚235123540991120 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚247168944993280 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 $ _timestamp‚1.788471026436136e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12032612682752 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 164695713 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚274006573194752 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚261769094575616 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11998492294108 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚247122050324588 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12032612682752 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12032612682752 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11998263313024 _runtime‚7079.621156829 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 280117282 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115421569 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚261769070635520 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚247168962996224 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 280143628 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚247168962996224 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12032379014144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69040648 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43817937 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12032379014144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69040650 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 164722057 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚259201575678976 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 112858587 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115421569 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚631 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚629 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚247168944993280 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 _step‚496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 164722057 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12237788984320 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259201324007424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚259201575678976 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115421571 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43817308 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚235123540991120 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚247168944993280 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 $ _timestamp‚1.788471026436136e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12032612682752 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 164695713 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚274006573194752 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚261769094575616 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11998492294108 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0ðKïö‚(:… òÍçÔ—´‚ˆ gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚116.916' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.229 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rÙ¾¼r÷‚(:ç òÍçÔº›ß‰ disk.sdb2.out‚ 774.55859375 disk.rbd0.out‚ 1.962890625 disk.sda.out‚ 774.55859375# disk.nvme0n1.out‚41415.40234375# proc.memory.rssMB‚ 3030.24609375 disk.sdb2.in‚ 4.48828125# disk.nvme1n1.in‚201199.94140625 disk.sda2.out‚ 774.55859375) proc.memory.percent‚18.49515438079834# disk.nvme1n1.out‚50237.93359375 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda.in‚ 4.40234375 disk.dm-0.in‚530976.41015625 disk.sda1.in‚0 disk.sda2.in‚ 4.40234375 disk.md0p1.in‚8.890625 disk.rbd2.in‚0 disk.sdb.out‚ 774.55859375) proc.memory.availableMB‚ 10472.2109375 proc.cpu.threads‚17 network.sent‚ 1.531792e+07# disk./.usageGB‚2974.80135345459 disk.rbd2.out‚ 1.962890625 network.recv‚ 2.373547e+06 disk.md0.in‚8.890625 disk.nvme0n1.in‚ 329776.46875 disk.md0p1.out‚ 746.765625 disk.sdb.in‚ 4.48828125 disk.dm-0.out‚ 91653.3359375 disk.sda1.out‚0$ memory_percent‚36.08269691467285) disk./.usagePercent‚83.17669089368215 disk.md0.out‚ 746.765625 disk.rbd0.in‚0.015625 cpu‚25.199990633676332&[52ø- _wandbruntime‚7080  _runtime‚7080M(—<ùòM XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚274050271828480 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12034343040000 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 112876177 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚261810754605568 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ZFDeviiÍwy¼"ceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚235161071348544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115439844 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 280188621 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12034343040000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 280162249 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12034576721920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚247208249033216 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚261810778545664 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚235161088387904 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 % _timestamp‚1.7884710275266109e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 trainer/global_step‚6049 _runtime‚7080.713698624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12000450783302 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12034576721920 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 _step‚497 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69051450 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 115439846 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚259242825755136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚247161293150554 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 112875544 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 115439844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚259242825755136 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 164722405 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚259242574070272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚633 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚274050586374656 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12239807828992 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚247161539171206 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚631 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 280162249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43824094 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115439846 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 164722405 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚247208231030272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 280188621 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 280188621 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚235161071348544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115439844 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 280188621 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12034343040000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 280162249 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12034576721920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚247208249033216 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚261810778545664 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚235161088387904 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 % _timestamp‚1.7884710275266109e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 trainer/global_step‚6049 _runtime‚7080.713698624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12000450783302 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12034576721920 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 _step‚497 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69051450 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 115439846 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚259242825755136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚247161293150554 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 112875544 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 115439844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚259242825755136 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 164722405 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚259242574070272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚633 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚274050586374656 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12239807828992 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚247161539171206 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚631 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 280162249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43824094 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115439846 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528ñõQÂ2û- _wandbruntime‚7080  _runtime‚7080õ ;}]ü× 1loss/FBt4/assignment_loss‚0.14294611669246088 _runtime‚7080.715105972 2loss/SRqqt4/assignment_loss‚2.0378412678837776 .loss/FRt4/detection_loss‚3.171488230643065 .loss/FBt2/detection_loss‚2.197970146482641 &loss/total_loss‚43.247556452506245 0loss/FRt1/assignment_loss‚2.9422397141655288 2loss/SRqqt2/assignment_loss‚1.7743235351266087 2loss/SRqqt3/assignment_loss‚0.9985305120698312 1loss/SRqqt2/detection_loss‚1.9767982218716595 0loss/SRqqt4/detection_loss‚2.963719031214714 .loss/FBt1/detection_loss‚2.461370848947101 _step‚498 2loss/SRqqt1/assignment_loss‚1.5580527346480537 1loss/FBt3/assignment_loss‚0.05011901599323393 trainer/global_step‚6049 0loss/FBt2/assignment_loss‚0.1533073798821957 /loss/FRt3/detection_loss‚1.4576996131996873 1loss/SRqqt1/detection_loss‚2.4177750566521206 epoch‚5 /loss/FRt4/assignment_loss‚2.990704834461212 /loss/FBt4/detection_loss‚2.9321344069072177 % _timestamp‚1.7884710275302036e+09 /loss/FBt3/detection_loss‚0.7079409230425353 0loss/FRt2/assignment_loss‚3.3149750462987204 1loss/SRqqt3/detection_loss‚0.7820923090415094 .loss/FRt1/detection_loss‚2.337277013808489 /loss/FRt2/detection_loss‚2.1086021771374033 0loss/FRt3/assignment_loss‚1.5616581498737065 1loss/FBt1/assignment_loss‚0.20799016646277677yÿKbýÜ /loss/FRt2/detection_loss‚2.1086021771374033 0loss/FRt3/assignment_loss‚1.5616581498737065 1loss/FBt1/assignment_loss‚0.20799016646277677 1loss/FBt4/assignment_loss‚0.14294611669246088 _runtime‚7080.715105972 2loss/SRqqt4/assignment_loss‚2.0378412678837776 .loss/FRt4/detection_loss‚3.171488230643065 .loss/FBt2/detection_loss‚2.197970146482641 &loss/total_loss‚43.247556452506245 0loss/FRt1/assignment_loss‚2.9422397141655288 2loss/SRqqt2/assignment_loss‚1.7743235351266087 2loss/SRqqt3/assignment_loss‚0.9985305120698312 1loss/SRqqt2/detection_loss‚1.9767982218716595 0loss/SRqqt4/detection_loss‚2.963719031214714 .loss/FBt1/detection_loss‚2.461370848947101 _step‚498 2loss/SRqqt1/assignment_loss‚1.5580527346480537 1loss/FBt3/assignment_loss‚0.05011901599323393 trainer/global_step‚6049 0loss/FBt2/assignment_loss‚0.1533073798821957 /loss/FRt3/detection_loss‚1.4576996131996873 1loss/SRqqt1/detection_loss‚2.4177750566521206 epoch‚5 /loss/FRt4/assignment_loss‚2.990704834461212 /loss/FBt4/detection_loss‚2.9321344069072177 % _timestamp‚1.7884710275302036e+09 /loss/FBt3/detection_loss‚0.7079409230425353 0loss/FRt2/assignment_loss‚3.3149750462987204 1loss/SRqqt3/detection_loss‚0.7820923090415094 .loss/FRt1/detection_loss‚2.337277013808489ò¾|¡‘þ‚(:† ÎçÔ¡©½‡ gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚140.754' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.1885 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Üìu±ÿ‚(:ö ÎçÔóà؈! disk.nvme1n1.out‚ 50238.015625 disk.sdb2.in‚ 4.48828125 disk.sdb.in‚ 4.48828125 disk.sda.out‚776.0830078125 disk.md0.in‚8.890625 disk.sdb1.out‚0 disk.md0p1.in‚8.890625% disk./.usageGB‚2974.8014221191406 disk.dm-0.in‚530976.41015625 disk.md0p1.out‚ 748.23046875 proc.cpu.threads‚17 network.sent‚ 1.5464317e+07 disk.sda2.out‚776.0830078125 disk.dm-0.out‚91919.36328125 disk.sda1.out‚0 disk.md0.out‚ 748.23046875 disk.rbd0.out‚ 1.962890625 disk.rbd2.in‚0 disk.sda.in‚ 4.40234375 disk.nvme0n1.in‚ 329776.46875 disk.sda2.in‚ 4.40234375$ memory_percent‚36.08601093292236# proc.memory.rssMB‚ 3030.24609375) proc.memory.percent‚18.49515438079834 network.recv‚ 2.380872e+06 disk.rbd0.in‚0.015625# disk.nvme1n1.in‚201199.94140625 cpu‚25.199647627594906) disk./.usagePercent‚83.17669281357173 disk.sdb2.out‚776.0830078125 disk.rbd2.out‚ 1.962890625 disk.sdb.out‚776.0830078125 disk.sdb1.in‚0# disk.nvme0n1.out‚41681.34765625* proc.memory.availableMB‚10471.66796875 disk.sda1.in‚0$:·€€‚(:õ ÎçÔ“߶‡ disk.sdb1.in‚0* proc.memory.availableMB‚10471.34765625 network.sent‚ 1.5550991e+07 network.recv‚ 2.387641e+06% disk./.usageGB‚2973.7666397094727 disk.md0p1.out‚ 749.66015625 disk.rbd0.in‚0.015625# disk.nvme1n1.in‚201199.94140625 disk.sdb.in‚ 4.48828125 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚ 1.962890625% memory_percent‚36.087965965270996# proc.memory.rssMB‚ 3030.24609375 disk.md0p1.in‚8.890625 disk.sdb2.out‚777.5576171875 disk.md0.in‚8.890625 disk.sdb1.out‚0 cpu‚25.19930780403017 disk.sdb2.in‚ 4.48828125 disk.rbd2.out‚ 1.962890625) disk./.usagePercent‚83.14775986430732 disk.dm-0.in‚530976.41015625 disk.dm-0.out‚92159.68359375 disk.sda2.out‚777.5576171875 disk.md0.out‚ 749.66015625 disk.sdb.out‚777.5576171875 disk.sda.out‚777.5576171875 disk.nvme0n1.in‚ 329776.46875 disk.nvme0n1.out‚ 41750.59375 disk.rbd2.in‚0) proc.memory.percent‚18.49515438079834 disk.sda.in‚ 4.40234375 disk.sda2.in‚ 4.40234375# disk.nvme1n1.out‚50409.08984375.!R=Ž‚(:ƒ ÎçÔˆ¶Ñ™ gpu.0.gpu‚43.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚137.36' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.34 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û‹ ~’‚‚(:‡ ŸÎçÔ۞LJ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚138.919' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.72975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n~°Î{ƒ‚(:ð ŸÎçÔ…Ë⇠network.recv‚ 2.391541e+06 disk.sda2.in‚ 4.40234375# disk.nvme1n1.out‚50409.16796875 disk.sdb.in‚ 4.48828125) proc.memory.percent‚18.49515438079834% disk./.usageGB‚2973.8934898376465 disk.sda2.out‚ 778.814453125 disk.sdb.out‚ 778.814453125) proc.memory.availableMB‚ 10471.3828125 disk.dm-0.out‚ 92324.6875 disk.sda1.in‚0 disk.sdb2.in‚ 4.48828125 disk.rbd2.in‚0# disk.nvme1n1.in‚201199.94140625 disk.sda.out‚ 778.814453125 disk.nvme0n1.in‚ 329776.46875 disk.md0p1.out‚ 750.85546875 disk.sdb1.in‚0 disk.dm-0.in‚530976.41015625 disk.sda1.out‚0 disk.md0.in‚8.890625 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 1.962890625 disk.sda.in‚ 4.40234375 cpu‚25.198922567795087) disk./.usagePercent‚83.15130664698196 disk.md0p1.in‚8.890625 disk.sdb1.out‚0# proc.memory.rssMB‚ 3030.24609375 proc.cpu.threads‚17 disk.sdb2.out‚ 778.814453125 disk.md0.out‚ 750.85546875 network.sent‚ 1.5556302e+07# disk.nvme0n1.out‚41915.51953125 disk.rbd0.out‚ 1.962890625% memory_percent‚36.087751388549805D_±=2„- _wandbruntime‚7131  _runtime‚7131!eÎHÇ…Á % _timestamp‚1.7884710783509574e+09 _runtime‚7131.535913185 _step‚499 & lr-AdamW/pg1‚0.0004774708696460329 & lr-AdamW/pg2‚0.0004774708696460329 trainer/global_step‚6099æ ÇÊÌ†Æ & lr-AdamW/pg1‚0.0004774708696460329 & lr-AdamW/pg2‚0.0004774708696460329 trainer/global_step‚6099 % _timestamp‚1.7884710783509574e+09 _runtime‚7131.535913185 _step‚499ó0((w2‡- _wandbruntime‚7131  _runtime‚7131D—NˆçO C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚261264093235200 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 329239552 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚249096506236296 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚249096260215852 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12096201353532 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 282368984 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚249133246842880 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12130828389376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12130594720768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884710783519619e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚276190742043648 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12130828389376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 7131.53oK”)+715624 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 166060009 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12130594720768 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 166060009 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 166033665 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚237000075901680 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚261264093235200 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚261263841563648 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 166033665 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 113742141 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44161391 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚276191071283200 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚237000058862320 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116335321 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 282395330 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚263853269137920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _step‚500 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 305299456 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12337472905728Ðþ´&ò'‰ìO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚261264093235200 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 329239552 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚249096506236296 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚249096260215852 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12096201353532 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 282368984 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚249133246842880 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12130828389376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12130594720768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884710783519619e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚276190742043648 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12130828389376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 7131.53715624 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 166060009 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12130594720768 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 166060009 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 166033665 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚237000075901680 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚261264093235200 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚261263841563648 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 166033665 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 113742141 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44161391 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚276191071283200 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚237000058862320 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116335321 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 282395330 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚263853269137920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _step‚500 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 305299456 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12337472905728 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273ôÂÀÄ?2Š- _wandbruntime‚7132  _runtime‚7132;w4ƒ÷&‹ñM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚249172532879872 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12339759950336 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116353596 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113759260 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12132792428544 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 282440323 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12098388823810 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12098159842518 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69591552 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116353594 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚249172532879872 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 113759940 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚276234418257920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 _runtime‚7132.483794677 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44167710 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 166086727 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚249172550882816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116353594 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12339465149952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚249135749062262 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚261305343311360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 166060357 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚6099 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚261305091626496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 166086727 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚276234736998400 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12132558746624 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12132792428544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 282413951 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚237037606259104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 282440323 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚237037589219744 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚263894977048064 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚249135995082914 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777®|_N—ŒöM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 166086727 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚249172550882816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116353594 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12339465149952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚249135749062262 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚261305343311360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 166060357 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚6099 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚261305091626496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 166086727 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚276234736998400 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12132558746624 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12132792428544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 282413951 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚237037606259104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 282440323 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚237037589219744 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚263894977048064 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚249135995082914 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚249172532879872 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12339759950336 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116353596 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113759260 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12132792428544 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 282440323 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12098388823810 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12098159842518 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69591552 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116353594 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚249172532879872 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 113759940 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚276234418257920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 23"( Ðe3681920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 _runtime‚7132.483794677 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44167710õ9®%2- _wandbruntime‚7132  _runtime‚7132ZÌEÓYŽÓ 0loss/FRt3/assignment_loss‚1.1352299972186013 1loss/FBt2/assignment_loss‚0.29478149871203385 0loss/SRqqt2/detection_loss‚2.397865852341056 %loss/total_loss‚43.60256272569188 _runtime‚7132.484825543 trainer/global_step‚6099 0loss/FRt2/assignment_loss‚3.6804712920477898 1loss/SRqqt1/assignment_loss‚1.556083871345771 2loss/SRqqt4/assignment_loss‚1.9618571691023998 /loss/FRt1/detection_loss‚3.0986526062091193 .loss/FRt4/detection_loss‚3.180294749637445 0loss/SRqqt4/detection_loss‚2.039701004823049 .loss/FBt1/detection_loss‚2.588835601623242 2loss/SRqqt2/assignment_loss‚1.7220790954306722 1loss/FBt1/assignment_loss‚0.49597571188439377 1loss/FBt4/assignment_loss‚0.13990141495361666 /loss/FRt3/detection_loss‚1.4875358158929481 /loss/FBt3/detection_loss‚0.6288080567257924 % _timestamp‚1.7884710793000817e+09 1loss/SRqqt3/detection_loss‚0.8343053434555437 .loss/FBt4/detection_loss‚2.505241354306539 epoch‚5 .loss/FRt1/assignment_loss‚2.60660428305467 0loss/FRt4/assignment_loss‚3.2294737050930657 1loss/SRqqt1/detection_loss‚2.1051740638519587 2loss/SRqqt3/assignment_loss‚1.1233646441205387 /loss/FRt2/detection_loss‚2.3464779870242416 _step‚502 1loss/FBt3/assignment_loss‚0.10054707845133923 -loss/FBt2/detection_loss‚2.34330052838606ô5 Y^Ø 1loss/SRqqt1/detection_loss‚2.1051740638519587 2loss/SRqqt3/assignment_loss‚1.1233646441205387 /loss/FRt2/detection_loss‚2.3464779870242416 _step‚502 1loss/FBt3/assignment_loss‚0.10054707845133923 -loss/FBt2/detection_loss‚2.34330052838606 0loss/FRt3/assignment_loss‚1.1352299972186013 1loss/FBt2/assignment_loss‚0.29478149871203385 0loss/SRqqt2/detection_loss‚2.397865852341056 %loss/total_loss‚43.60256272569188 _runtime‚7132.484825543 trainer/global_step‚6099 0loss/FRt2/assignment_loss‚3.6804712920477898 1loss/SRqqt1/assignment_loss‚1.556083871345771 2loss/SRqqt4/assignment_loss‚1.9618571691023998 /loss/FRt1/detection_loss‚3.0986526062091193 .loss/FRt4/detection_loss‚3.180294749637445 0loss/SRqqt4/detection_loss‚2.039701004823049 .loss/FBt1/detection_loss‚2.588835601623242 2loss/SRqqt2/assignment_loss‚1.7220790954306722 1loss/FBt1/assignment_loss‚0.49597571188439377 1loss/FBt4/assignment_loss‚0.13990141495361666 /loss/FRt3/detection_loss‚1.4875358158929481 /loss/FBt3/detection_loss‚0.6288080567257924 % _timestamp‚1.7884710793000817e+09 1loss/SRqqt3/detection_loss‚0.8343053434555437 .loss/FBt4/detection_loss‚2.505241354306539 epoch‚5 .loss/FRt1/assignment_loss‚2.60660428305467 0loss/FRt4/assignment_loss‚3.2294737050930657öÁÇb{‚(:ð ®ÎçÔ‚Á‡! disk.sdb2.out‚780.96240234375 disk.nvme0n1.in‚ 329776.46875! disk.nvme0n1.out‚ 42068.359375 disk.md0p1.in‚8.890625* proc.memory.availableMB‚10469.54296875 cpu‚25.19854339548037 network.sent‚ 1.5560419e+07 disk.rbd2.in‚0 disk.sdb.out‚780.96240234375 disk.sda1.in‚0 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 1.962890625 disk.dm-0.in‚530976.41015625! disk.sda2.out‚780.96240234375 disk.sdb2.in‚ 4.48828125 disk.sdb1.out‚0 disk.md0p1.out‚752.9375) proc.memory.percent‚18.49515438079834 disk.sdb1.in‚0 disk.sda2.in‚ 4.40234375$ memory_percent‚36.09898090362549# proc.memory.rssMB‚ 3030.24609375) disk./.usagePercent‚83.15479113990585 disk.rbd2.out‚ 1.962890625# disk.nvme1n1.in‚201199.94140625 disk.sdb.in‚ 4.48828125 disk.dm-0.out‚92477.81640625 disk.sda1.out‚0 proc.cpu.threads‚17 disk.md0.in‚8.890625 disk.md0.out‚752.9375$ disk./.usageGB‚2974.018112182617# disk.nvme1n1.out‚50409.45703125 disk.sda.in‚ 4.40234375 disk.sda.out‚780.96240234375 network.recv‚ 2.396228e+06Mvöœ‘‚(:‘ ®Îçԥѳ‡ gpu.0.gpu‚14.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.505' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚27.126249999999995 gpu.0.smClock‚1215 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%¡ -’’‚(:‡ ½ÎçÔØ·Ó… gpu.0.gpu‚19.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚142.313' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.57825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@œS9y“‚(:î ½ÎçÔ¹©“‡ disk.sda.out‚782.4541015625* proc.memory.availableMB‚10468.99609375 disk.sdb2.out‚782.4541015625 disk.md0.in‚8.890625 disk.rbd0.in‚0.015625 disk.sdb1.out‚0 disk.nvme0n1.in‚ 329776.46875 disk.dm-0.out‚ 92559.4921875 disk.md0p1.out‚ 754.34375 proc.cpu.threads‚17 network.recv‚ 2.402839e+06$ memory_percent‚36.10231876373291 disk.rbd2.in‚0 disk.sdb.out‚782.4541015625 disk.sda2.in‚ 4.40234375# proc.memory.rssMB‚ 3030.24609375 disk.sdb.in‚ 4.48828125 disk.dm-0.in‚530976.41015625 disk.sda1.out‚0) proc.memory.percent‚18.49515438079834 network.sent‚ 1.5670345e+07) disk./.usagePercent‚83.15155964576417 disk.rbd0.out‚ 1.962890625 disk.sda.in‚ 4.40234375 disk.sda1.in‚0 disk.sda2.out‚782.4541015625 cpu‚25.198166458163822" disk.nvme0n1.out‚ 42149.6953125 disk.md0p1.in‚8.890625% disk./.usageGB‚2973.9025382995605# disk.nvme1n1.in‚201199.94140625! disk.nvme1n1.out‚ 50409.796875 disk.sdb1.in‚0 disk.sdb2.in‚ 4.48828125 disk.md0.out‚ 754.34375 disk.rbd2.out‚ 1.962890625½t”‚(:é ÌÎçÔ₯‡% disk./.usageGB‚2974.0706329345703 disk.sdb1.out‚0% memory_percent‚36.101436614990234# proc.memory.rssMB‚ 3030.24609375 disk.sda.out‚783.9892578125 disk.md0.out‚755.8125 network.recv‚ 2.41395e+06# disk.nvme1n1.in‚201199.94140625# disk.nvme1n1.out‚50496.02734375 disk.sda1.in‚0 disk.sda2.in‚ 4.40234375 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 1.962890625( disk./.usagePercent‚83.1562596421121 disk.sdb.in‚ 4.48828125 disk.nvme0n1.in‚ 329776.46875 disk.sda.in‚ 4.40234375 disk.dm-0.in‚530976.41015625 disk.sdb2.in‚ 4.48828125 disk.rbd2.out‚ 1.962890625 cpu‚25.19782162143937 disk.sdb1.in‚0# disk.nvme0n1.out‚42273.12890625 disk.dm-0.out‚ 92769.15625 disk.sda1.out‚0 disk.rbd2.in‚0) proc.memory.percent‚18.49515438079834 network.sent‚ 1.5680017e+07 disk.sda2.out‚783.9892578125 disk.md0p1.in‚8.890625 disk.md0.in‚8.890625( proc.memory.availableMB‚ 10469.140625 proc.cpu.threads‚17 disk.sdb.out‚783.9892578125 disk.md0p1.out‚755.8125 disk.sdb2.out‚783.9892578125[Ì’•‚(:‡ ÌÎçÔäÀå‡ gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚143.263' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.81575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯ÜR|–‚(:ñ ÛÎçÔ›Âú† disk.dm-0.out‚ 92898.0546875 disk.sda1.in‚0 disk.md0p1.in‚8.890625# disk.nvme1n1.in‚201199.94140625 cpu‚25.197448176942387# disk.nvme1n1.out‚50582.06640625 disk.sdb1.in‚0) disk./.usagePercent‚83.14856141820623 disk.sda1.out‚0 disk.rbd0.out‚ 1.962890625 disk.md0.out‚ 757.04296875) proc.memory.availableMB‚ 10469.1015625 disk.md0p1.out‚ 757.04296875 disk.sdb2.in‚ 4.48828125 disk.sdb.in‚ 4.48828125 disk.sdb1.out‚0 disk.sda.in‚ 4.40234375# proc.memory.rssMB‚ 3030.24609375$ disk./.usageGB‚2973.795307159424 disk.sda2.in‚ 4.40234375 disk.sdb.out‚ 785.275390625# disk.nvme0n1.out‚42315.98828125 disk.dm-0.in‚530976.41015625 disk.sdb2.out‚ 785.275390625 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.rbd2.out‚ 1.962890625 disk.nvme0n1.in‚ 329776.46875 network.recv‚ 2.417542e+06 disk.sda2.out‚ 785.275390625 disk.sda.out‚ 785.275390625 network.sent‚ 1.568532e+07 disk.md0.in‚8.890625% memory_percent‚36.101675033569336) proc.memory.percent‚18.49515438079834 proc.cpu.threads‚17õ™‘—‚(:† ÛÎçÔÈ°‰‡ gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚138.302' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.5755 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶%‚ 2˜- _wandbruntime‚7185  _runtime‚7185‡é÷əà ' lr-AdamW/pg1‚0.00047158832823576436 ' lr-AdamW/pg2‚0.00047158832823576436 trainer/global_step‚6149 % _timestamp‚1.7884711321583118e+09 _runtime‚7185.343380365 _step‚503Lb—=ÎšÈ ' lr-AdamW/pg1‚0.00047158832823576436 ' lr-AdamW/pg2‚0.00047158832823576436 trainer/global_step‚6149 % _timestamp‚1.7884711321583118e+09 _runtime‚7185.343380365 _step‚503÷ÂRê2›- _wandbruntime‚7185  _runtime‚7185Ô’@î'œèO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 117249071 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44507821 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 117249069 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12228810427392 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117249069 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 167397961 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚251097548692480 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 114627982 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚251070962148004 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚263326359119872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 _runtime‚7185.344123309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 114628671 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70120850 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70120848 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 167371617 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12229044096000 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚278374778689536 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚263326610791424 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚251070716127560 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44507134 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚263326359119872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚265937467640320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457Ž&¯ó'íO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 117249071 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44507821 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 117249069 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12228810427392 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117249069 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 167397961 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚251097548692480 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 114627982 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚251070962148004 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚263326359119872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 _runtime‚7185.344123309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 114628671 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70120850 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70120848 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 167371617 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12229044096000 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚278374778689536 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚263326610791424 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚251070716127560 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44507134 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚263326359119872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚265937467640320ø.×2ž- _wandbruntime‚7186  _runtime‚7186Vú/ãŸòM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12196097883026 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 % _timestamp‚1.7884711331030867e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_poòDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 284692025 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚70131652 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚505 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 167424679 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 trainer/global_step‚6149 _runtime‚7186.288686618 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚263367860867584 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚251136852732416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚265979175550464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 167398309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚263367609182720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚677 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚238914124130304 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚251136834729472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 117267344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44513803 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70131650 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 167398309 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 117267346 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚251136852732416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44514480 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114645453 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 284665653 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚278418770668544 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚251110450994622 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚263367860867584 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12230774453248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 284692025 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117267346 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 167424679 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12231008135168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 167398309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚263367609182720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚677 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚238914124130304 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚251136834729472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 117267344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44513803 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70131650 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 167398309 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 117267346 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚251136852732416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44514480 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114645453 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 284665653 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚278418770668544 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚251110450994622 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚263367860867584 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12230774453248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 284692025 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117267346 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 167424679 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12231008135168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 284692025 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚70131652 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚505 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 167424679 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 trainer/global_step‚6149 _runtime‚7186.288686618 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚263367860867584 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚251136852732416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚265979175550464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0ù ,Ê2¡- _wandbruntime‚7186  _runtime‚7186‚òýZ¢Ô % _timestamp‚1.7884711331055727e+09 /loss/FRt1/assignment_loss‚2.419526787267791 %loss/total_loss‚42.17072210748085 0loss/SRqqt1/detection_loss‚2.045956925912337 trainer/global_step‚6149 _runtime‚7186.290279796 _step‚506 epoch‚5 2loss/SRqqt2/assignment_loss‚1.7063936480591375 1loss/FBt4/assignment_loss‚0.17463682886168766 1loss/SRqqt4/detection_loss‚2.4450063705444336 /loss/FBt3/detection_loss‚0.6606880151205432 /loss/FRt4/assignment_loss‚3.346025366783142 0loss/FBt1/assignment_loss‚0.1846665951802997 1loss/FBt2/assignment_loss‚0.17836931459314087 1loss/FBt3/assignment_loss‚0.11322845528974419 /loss/FRt4/detection_loss‚3.1612062740325926 /loss/FBt4/detection_loss‚1.9921450001351975 1loss/SRqqt1/assignment_loss‚1.617928610606627 1loss/SRqqt3/detection_loss‚0.9220144896368895 -loss/FBt2/detection_loss‚2.36328530497849 0loss/FRt2/assignment_loss‚3.6232284565229675 2loss/SRqqt3/assignment_loss‚1.1823547231199751 0loss/SRqqt2/detection_loss‚2.507407142269996 2loss/SRqqt4/assignment_loss‚1.4276381931625879 /loss/FRt1/detection_loss‚2.6129232197999954 /loss/FRt3/detection_loss‚1.3256409879773856 /loss/FBt1/detection_loss‚2.1177943721413612 0loss/FRt3/assignment_loss‚1.6075904987752438 .loss/FRt2/detection_loss‚2.435066526709282ëmô_£Ù /loss/FBt3/detection_loss‚0.6606880151205432 /loss/FRt4/assignment_loss‚3.346025366783142 0loss/FBt1/assignment_loss‚0.1846665951802997 1loss/FBt2/assignment_loss‚0.17836931459314087 1loss/FBt3/assignment_loss‚0.11322845528974419 /loss/FRt4/detection_loss‚3.1612062740325926 /loss/FBt4/detection_loss‚1.9921450001351975 1loss/SRqqt1/assignment_loss‚1.617928610606627 1loss/SRqqt3/detection_loss‚0.9220144896368895 -loss/FBt2/detection_loss‚2.36328530497849 0loss/FRt2/assignment_loss‚3.6232284565229675 2loss/SRqqt3/assignment_loss‚1.1823547231199751 0loss/SRqqt2/detection_loss‚2.507407142269996 2loss/SRqqt4/assignment_loss‚1.4276381931625879 /loss/FRt1/detection_loss‚2.6129232197999954 /loss/FRt3/detection_loss‚1.3256409879773856 /loss/FBt1/detection_loss‚2.1177943721413612 0loss/FRt3/assignment_loss‚1.6075904987752438 .loss/FRt2/detection_loss‚2.435066526709282 % _timestamp‚1.7884711331055727e+09 /loss/FRt1/assignment_loss‚2.419526787267791 %loss/total_loss‚42.17072210748085 0loss/SRqqt1/detection_loss‚2.045956925912337 trainer/global_step‚6149 _runtime‚7186.290279796 _step‚506 epoch‚5 2loss/SRqqt2/assignment_loss‚1.7063936480591375 1loss/FBt4/assignment_loss‚0.17463682886168766 1loss/SRqqt4/detection_loss‚2.4450063705444336ú¥³Q¬’¤‚(:‡ êÎçÔ’„›ˆ gpu.0.gpu‚26.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚139.251' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.81275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿtü‚¥‚(:÷ êÎçÔŸž‹‰ disk.sdb1.out‚0 disk.sdb2.in‚ 4.48828125 disk.md0.in‚8.890625 disk.nvme0n1.out‚ 42316.28125 disk.sda2.in‚ 4.40234375 disk.md0.out‚ 759.0234375 network.sent‚ 1.5689562e+07 disk.rbd0.in‚0.015625 disk.nvme0n1.in‚ 329776.46875$ memory_percent‚36.10208034515381# proc.memory.rssMB‚ 3030.24609375% disk./.usageGB‚2973.8822631835938 disk.dm-0.out‚92904.70703125 disk.sda1.out‚0 disk.md0p1.in‚8.890625 cpu‚25.197126936000497 proc.cpu.threads‚17 disk.rbd2.out‚ 1.962890625 disk.sdb.out‚787.30712890625 disk.sda.in‚ 4.40234375 disk.md0p1.out‚ 759.0234375 network.recv‚ 2.422018e+06) disk./.usagePercent‚83.15099274503594 disk.rbd0.out‚ 1.962890625 disk.rbd2.in‚0 disk.sdb1.in‚0 disk.sda.out‚787.30712890625 disk.dm-0.in‚530976.41015625 disk.sda1.in‚0! disk.sda2.out‚787.30712890625! disk.sdb2.out‚787.30712890625* proc.memory.availableMB‚10469.03515625) proc.memory.percent‚18.49515438079834# disk.nvme1n1.in‚201199.94140625# disk.nvme1n1.out‚50588.42578125 disk.sdb.in‚ 4.48828125]ÃBÿ¦‚(:… ùÎçÔΨ¿‡ gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚134.908' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.727 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:"uö…§‚(:ú ùÎçÔ„ÄÛ‡% memory_percent‚37.148499488830566 network.recv‚ 2.428585e+06% disk./.usageGB‚2974.3284759521484 disk.nvme0n1.in‚ 329776.46875 disk.sda2.in‚ 4.40234375 cpu‚25.196763442171097 disk.md0p1.in‚8.890625 disk.md0.in‚8.890625 disk.rbd0.out‚ 1.962890625 disk.sda.in‚ 4.40234375 disk.md0p1.out‚ 760.56640625# proc.memory.rssMB‚ 3030.24609375 network.sent‚ 1.5799406e+07 disk.sdb2.in‚ 4.48828125# disk.nvme1n1.in‚201199.94140625 disk.sdb.in‚ 4.48828125) disk./.usagePercent‚83.16346904079927 disk.rbd2.out‚ 1.962890625 disk.sdb1.in‚0 disk.sda1.out‚0* proc.memory.availableMB‚10297.58984375 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.rbd0.in‚0.015625 disk.sdb1.out‚0) proc.memory.percent‚18.49515438079834# disk.nvme1n1.out‚50859.82421875 disk.sdb.out‚788.91259765625 disk.sda.out‚788.91259765625# disk.nvme0n1.out‚42316.33203125 disk.dm-0.in‚530976.41015625 disk.dm-0.out‚ 93176.15625 disk.sda1.in‚0! disk.sda2.out‚788.91259765625! disk.sdb2.out‚788.91259765625 disk.md0.out‚ 760.56640625m‚£¤¨‚(:„ ˆÏçÔ×· gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚96.756' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚24.189 gpu.0.smClock‚1260 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ôs÷{©‚(:ð ˆÏçÔ¯ãøˆ! disk.nvme1n1.out‚ 50913.953125 network.sent‚ 1.580536e+07 disk.sdb2.out‚ 790.466796875 disk.md0.out‚ 762.05078125 disk.rbd0.out‚ 1.962890625 disk.sda.out‚ 790.466796875 proc.cpu.threads‚17 disk.sda2.out‚ 790.466796875 disk.rbd2.out‚ 1.962890625 disk.nvme0n1.in‚ 329776.46875 disk.sda1.in‚0 disk.sda2.in‚ 4.40234375 disk.md0p1.out‚ 762.05078125 cpu‚25.19641703177754# disk.nvme1n1.in‚201199.94140625 disk.sdb2.in‚ 4.48828125 disk.sda.in‚ 4.40234375 disk.dm-0.out‚93230.53515625 disk.md0p1.in‚8.890625 disk.sdb1.in‚0 network.recv‚ 2.432824e+06* proc.memory.availableMB‚10297.58203125# proc.memory.rssMB‚ 3030.24609375 disk.md0.in‚8.890625 disk.sdb.in‚ 4.48828125 disk.dm-0.in‚530976.41015625 disk.sda1.out‚0$ memory_percent‚37.14854717254639) disk./.usagePercent‚83.16521251385734% disk./.usageGB‚2974.3908309936523 disk.sdb1.out‚0# disk.nvme0n1.out‚42316.58203125) proc.memory.percent‚18.49515438079834 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.sdb.out‚ 790.466796875aòþ2ª- _wandbruntime‚7241  _runtime‚7241Ñ1”Ç«Á % _timestamp‚1.7884711880278528e+09 _runtime‚7241.212837366 _step‚507 & lr-AdamW/pg1‚0.0004656934158784286 & lr-AdamW/pg2‚0.0004656934158784286 trainer/global_step‚6199y±Ø$Ì¬Æ & lr-AdamW/pg1‚0.0004656934158784286 & lr-AdamW/pg2‚0.0004656934158784286 trainer/global_step‚6199 % _timestamp‚1.7884711880278528e+09 _runtime‚7241.212837366 _step‚507ût ˜ä2­- _wandbruntime‚7241  _runtime‚7241jr§¤i®çO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 115515497 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44854547 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚253061850542080 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118162819 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚253045172039268 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚268021690082816 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12537072868352 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12292077434548 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚253061850542080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚265388876676096 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 286898734 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 168709569 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚280559051473408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMoYxà(„nitor.on_train_batch_start/inactive_split.large_pool.freed‚70660948 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 286872388 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚657 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _step‚508 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 168735913 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚265389128347648 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12327026134016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70660950 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚655 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚253061868545024 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118162821 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚240753094604720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚240753111644080 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12537361390592 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12292306415632 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 115514840 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118162819 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚265388876676096 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚253061868545024 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚240753111644080 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12537361390592 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12292306415632 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 115514840 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118162819 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚265388876676096 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚253061868545024 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚253061850542080 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118162819 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚253045172039268 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚268021690082816 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12537072868352 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12292077434548 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚253061850542080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚265388876676096üû”Ë42°- _wandbruntime‚7242  _runtime‚7242¸÷ÖÖø&±òM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 286917355 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12539355010560 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 168736261 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚253084906906330 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301963264 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12539076987392 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12329223841792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12294035923534 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _step‚509 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 168762631 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44860903 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚280602729063424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118181096 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚253101154582016 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚653 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚7242.248447049 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚240790642001504 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 278023168 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚240790624962144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12329223841792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚655 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12294264904826 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚265430126738944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12328990159872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚253101154582016 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚253101136579072 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115532653 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44861556 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 trainer/global_step‚6199 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 % _timestamp‚1.7884711890627136e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚265430378423808 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 115533308 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚268063350112768 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚265430378423808 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118181096 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚253084660885678 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 286943727 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0xlÖa=²÷M YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12328990159872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚253101154582016 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚253101136579072 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115532653 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44861556 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 trainer/global_step‚6199 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 % _timestamp‚1.7884711890627136e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚265430378423808 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 115533308 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚268063350112768 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚265430378423808 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118181096 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚253084660885678 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 286943727 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 286917355 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12539355010560 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 168736261 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚253084906906330 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301963264 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12539076987392 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12329223841792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12294035923534 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _step‚509 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 168762631 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44860903 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚280602729063424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118181096 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚253101154582016 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚653 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚7242.248447049 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚240790642001504 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 278023168 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚240790624962144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12329223841792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚655 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12294264904826 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚265430126738944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104ýãR~2³- _wandbruntime‚7242  _runtime‚7242±¯Ï^´Ø 2loss/SRqqt2/assignment_loss‚1.6716446298541445 1loss/FBt3/assignment_loss‚0.08209473471256587 _step‚510 0loss/FRt3/assignment_loss‚1.0989161363014808 /loss/FRt4/detection_loss‚3.4278101523717246 1loss/FBt2/assignment_loss‚0.23706757682860674 % _timestamp‚1.7884711890647068e+09 0loss/FRt4/assignment_loss‚3.9119425217310586 2loss/SRqqt3/assignment_loss‚0.8503734667884927 1loss/FBt4/assignment_loss‚0.23597720663353053 -loss/FRt1/detection_loss‚2.81567445397377 /loss/FRt2/detection_loss‚1.9427032182871733 /loss/FRt3/detection_loss‚1.6570080579473423 1loss/SRqqt3/detection_loss‚0.7710456100283462 /loss/FBt2/detection_loss‚1.9443425913651784 epoch‚5 1loss/FBt1/assignment_loss‚0.23920039804067983 1loss/SRqqt1/detection_loss‚2.0918657332658768 1loss/SRqqt4/detection_loss‚2.5602323988388322 .loss/FBt3/detection_loss‚0.779998782198303 .loss/FBt4/detection_loss‚2.577688604593277 %loss/total_loss‚43.98090508075197 0loss/FRt2/assignment_loss‚3.7973197451177634 2loss/SRqqt4/assignment_loss‚1.8887149557985108 0loss/SRqqt2/detection_loss‚2.147140736832763 0loss/FRt1/assignment_loss‚2.9732757528622944 2loss/SRqqt1/assignment_loss‚1.4567203006396692 /loss/FBt1/detection_loss‚2.8221473157405854 trainer/global_step‚6199 _runtime‚7242.249500394*[cµÝ 0loss/FRt4/assignment_loss‚3.9119425217310586 2loss/SRqqt3/assignment_loss‚0.8503734667884927 1loss/FBt4/assignment_loss‚0.23597720663353053 -loss/FRt1/detection_loss‚2.81567445397377 /loss/FRt2/detection_loss‚1.9427032182871733 /loss/FRt3/detection_loss‚1.6570080579473423 1loss/SRqqt3/detection_loss‚0.7710456100283462 /loss/FBt2/detection_loss‚1.9443425913651784 epoch‚5 1loss/FBt1/assignment_loss‚0.23920039804067983 1loss/SRqqt1/detection_loss‚2.0918657332658768 1loss/SRqqt4/detection_loss‚2.5602323988388322 .loss/FBt3/detection_loss‚0.779998782198303 .loss/FBt4/detection_loss‚2.577688604593277 %loss/total_loss‚43.98090508075197 0loss/FRt2/assignment_loss‚3.7973197451177634 2loss/SRqqt4/assignment_loss‚1.8887149557985108 0loss/SRqqt2/detection_loss‚2.147140736832763 0loss/FRt1/assignment_loss‚2.9732757528622944 2loss/SRqqt1/assignment_loss‚1.4567203006396692 /loss/FBt1/detection_loss‚2.8221473157405854 trainer/global_step‚6199 _runtime‚7242.249500394 2loss/SRqqt2/assignment_loss‚1.6716446298541445 1loss/FBt3/assignment_loss‚0.08209473471256587 _step‚510 0loss/FRt3/assignment_loss‚1.0989161363014808 /loss/FRt4/detection_loss‚3.4278101523717246 1loss/FBt2/assignment_loss‚0.23706757682860674 % _timestamp‚1.7884711890647068e+09þÓÝè"¶‚(:… —ÏçÔˆÚć gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚131.45' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.8625 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a=³‚·‚(:÷ —ÏçÔŽçôˆ disk.sda1.out‚0) proc.memory.percent‚18.49515438079834 disk.rbd2.in‚0# disk.nvme1n1.in‚201199.94140625 cpu‚25.196081089893767 disk.md0.out‚ 763.96484375) disk./.usagePercent‚83.18446164674988 disk.sda1.in‚0 disk.md0p1.in‚8.890625$ disk./.usageGB‚2975.079273223877 disk.sda2.in‚ 4.40234375 disk.rbd0.in‚0.015625# disk.nvme1n1.out‚51323.86328125 disk.sdb.in‚ 4.48828125$ memory_percent‚37.15050220489502# proc.memory.rssMB‚ 3030.24609375 disk.rbd2.out‚ 1.962890625 disk.sdb1.out‚0 disk.sda.in‚ 4.40234375 disk.sda.out‚792.44580078125! disk.nvme0n1.out‚ 42316.640625 network.sent‚ 1.580912e+07 disk.dm-0.out‚93640.50390625 disk.nvme0n1.in‚ 329776.46875* proc.memory.availableMB‚10297.26171875 disk.rbd0.out‚ 1.962890625 disk.sdb.out‚792.44580078125 disk.sdb1.in‚0 disk.sdb2.in‚ 4.48828125 network.recv‚ 2.43698e+06 disk.dm-0.in‚530976.41015625! disk.sda2.out‚792.44580078125 disk.md0p1.out‚ 763.96484375! disk.sdb2.out‚792.44580078125 disk.md0.in‚8.890625 proc.cpu.threads‚17-Á‚‘¸‚(:† ¦ÏçԵ˔‡ gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.966' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.9915 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BÇ7}|¹‚(:ñ ¦ÏçÔ՗ȇ$ disk./.usageGB‚2975.330265045166 disk.sdb2.out‚794.1845703125 disk.rbd2.in‚0$ memory_percent‚37.15355396270752 network.recv‚ 2.450395e+06 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 1.962890625 disk.sdb1.out‚0 disk.sda1.in‚0 disk.md0p1.in‚8.890625) proc.memory.percent‚18.49515438079834# disk.nvme1n1.out‚51376.12109375# disk.nvme0n1.out‚42316.94140625 disk.sdb2.in‚ 4.48828125 disk.sda.in‚ 4.40234375 disk.sda1.out‚0 disk.md0p1.out‚ 765.6484375 proc.cpu.threads‚17 network.sent‚ 1.5920397e+07 disk.sdb1.in‚0 disk.sda.out‚794.1845703125 disk.sda2.out‚794.1845703125 disk.md0.in‚8.890625 disk.sdb.out‚794.1845703125 disk.dm-0.in‚530976.41015625 disk.md0.out‚ 765.6484375 disk.rbd2.out‚ 1.962890625# disk.nvme1n1.in‚201199.94140625 disk.sdb.in‚ 4.48828125 disk.nvme0n1.in‚ 329776.46875 disk.sda2.in‚ 4.40234375 cpu‚25.195721877722278 disk.dm-0.out‚ 93693.0625* proc.memory.availableMB‚10296.76171875# proc.memory.rssMB‚ 3030.24609375) disk./.usagePercent‚83.19147948312136ÂKù ›º‚(: µÏçÔü阇 gpu.0.gpu‚8.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚104.161' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚26.040249999999997 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0D©"a»‚(:ô µÏçÔû›õ‡ disk.sda1.out‚0 disk.rbd2.in‚0 disk.sdb.in‚ 4.48828125 disk.sdb1.in‚0 disk.nvme0n1.in‚ 329776.46875 cpu‚25.195386034047907 disk.rbd0.out‚ 1.962890625$ memory_percent‚37.13850975036621 network.sent‚ 1.5927625e+07 network.recv‚ 2.454081e+06 disk.md0p1.in‚8.890625 disk.rbd2.out‚ 1.962890625# disk.nvme1n1.out‚51730.54296875# proc.memory.rssMB‚ 3030.24609375% disk./.usageGB‚2975.3274154663086 disk.sda1.in‚0 disk.sda2.in‚ 4.40234375 disk.sda2.out‚795.6357421875 disk.sdb2.in‚ 4.48828125 disk.sdb1.out‚0 disk.dm-0.out‚94047.53515625 disk.sdb.out‚795.6357421875 disk.sda.in‚ 4.40234375 disk.md0p1.out‚ 767.0390625 disk.dm-0.in‚530976.41015625) proc.memory.availableMB‚ 10299.2265625) proc.memory.percent‚18.49515438079834 proc.cpu.threads‚17 disk.sda.out‚795.6357421875" disk.nvme0n1.out‚ 42316.9921875) disk./.usagePercent‚83.19139980770386 disk.sdb2.out‚795.6357421875 disk.md0.in‚8.890625 disk.md0.out‚ 767.0390625 disk.rbd0.in‚0.015625# disk.nvme1n1.in‚201199.94140625õ-å•‘¼‚(:† ÄÏçÔß—Ó‡ gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚129.282' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.3205 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0È<öI~½‚(:ó ÄÏçÔ¹¹…ˆ# disk.nvme1n1.in‚201199.94140625 network.recv‚ 2.459162e+06 disk.sdb1.in‚0 disk.sda.in‚ 4.40234375 disk.md0p1.out‚ 768.96875 disk.rbd2.out‚ 1.962890625 cpu‚25.195053181882297 disk.sda.out‚797.62841796875! disk.sda2.out‚797.62841796875 disk.sdb2.in‚ 4.48828125! disk.sdb2.out‚797.62841796875 disk.md0.in‚8.890625$ memory_percent‚37.13846206665039) proc.memory.percent‚18.49515438079834 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0.out‚ 768.96875 disk.sdb.in‚ 4.48828125 network.sent‚ 1.5932444e+07" disk.nvme0n1.out‚ 42317.1015625 disk.dm-0.in‚530976.41015625 disk.sda2.in‚ 4.40234375 disk.rbd0.in‚0.015625# proc.memory.rssMB‚ 3030.24609375 proc.cpu.threads‚17) disk./.usagePercent‚83.19139820779587 disk.nvme0n1.in‚ 329776.46875 disk.dm-0.out‚94391.24609375 disk.rbd0.out‚ 1.962890625# disk.nvme1n1.out‚52074.14453125 disk.sdb.out‚797.62841796875 disk.sdb1.out‚0( proc.memory.availableMB‚ 10299.234375% disk./.usageGB‚2975.3273582458496 disk.md0p1.in‚8.890625 disk.rbd2.in‚0ƒÃF2¾- _wandbruntime‚7297  _runtime‚7297l¬ÝÉ¿Ã ' lr-AdamW/pg2‚0.00045978736106492554 trainer/global_step‚6249 % _timestamp‚1.7884712439204752e+09 _runtime‚7297.105516701 _step‚511 ' lr-AdamW/pg1‚0.00045978736106492554íP[ùÎÀÈ ' lr-AdamW/pg1‚0.00045978736106492554 ' lr-AdamW/pg2‚0.00045978736106492554 trainer/global_step‚6249 % _timestamp‚1.7884712439204752e+09 _runtime‚7297.105516701 _step‚511ÿ—vÑ€2Á- _wandbruntime‚7297  _runtime‚7297â=Rî'ÂèO BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚670 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚270105888585216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12425241840640 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚270105864645120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 170047521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306170880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 289124090 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 170073865 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚255026170394624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚255026152391680 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 116400843 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12390244456140 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119076571 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12637211658240 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 289150436 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚267451645903872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _step‚512 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚7297.107065728 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119076569 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚282742794072576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚267451394232320 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚242629629515280 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267451394232320 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚672 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282230784 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12425475509248 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 116401515 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119076569 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12390015475056 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 170073865 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45200465 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12636929427456 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚255019873971420 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71201048 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119076571 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚267451645903872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884712439218614e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45199795 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚255026152391680 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344$ÎÄáó'ÃíO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12425241840640 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚270105864645120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 170047521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306170880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 289124090 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 170073865 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚255026170394624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚255026152391680 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 116400843 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12390244456140 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119076571 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12637211658240 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 289150436 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚267451645903872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _step‚512 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚7297.107065728 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119076569 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚282742794072576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚267451394232320 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚242629629515280 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267451394232320 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚672 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282230784 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12425475509248 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 116401515 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119076569 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12390015475056 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 170073865 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45200465 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12636929427456 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚255019873971420 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71201048 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119076571 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚267451645903872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884712439218614e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45199795 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚255026152391680 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚670 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚270105888585216€»‚Z³2Ä- _wandbruntime‚7298  _runtime‚7298{¥ékÅòM QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 TEDeviceStatsMonitor.on_train_batch_end/requ]°óŒ%ested_bytes.large_pool.peak‚ 6236507136 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12639212871680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚255059116797386 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚267492895980032 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 170100583 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116418348 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 170074213 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚255065438428672 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚267492895980032 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚255059362818038 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _step‚513 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 289169057 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚242667159872704 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚255065456431616 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚282786785426944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 % _timestamp‚1.7884712450296638e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119094846 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚242667142833344 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12427205866496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚270147548615168 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚270147572555264 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 289195429 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71211850 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45206498 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119094844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119094844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12427205866496,h—ý&Æ÷M H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 289195429 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 289195429 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71211850 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45206498 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119094844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119094844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12427205866496 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12639212871680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚255059116797386 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚267492895980032 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 170100583 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116418348 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 170074213 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚255065438428672 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚267492895980032 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚255059362818038 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _step‚513 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 289169057 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚242667159872704 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚255065456431616 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚282786785426944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 % _timestamp‚1.7884712450296638e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119094846 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚242667142833344 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12427205866496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚270147548615168 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚270147572555264 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚256990472244224 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _runtime‚7348.513242225 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 291375792 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚269514163460096 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚269513911788544 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚256994083862684 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚269513911788544 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 171385473 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 117285577 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚284926793644032 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 _step‚516 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12523457547264 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45544429 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚244506130347120 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12523457547264 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119990321 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 291402138 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚256990472244224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 % _timestamp‚1.7884712953281949e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12523691215872 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚269514163460096 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚272190063147520 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 îó'ÕíO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119990319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71741150 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚671 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119990321 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 117286250 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚272190087087616 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 291375792 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚269514163460096 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚269513911788544 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚256994083862684 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚269513911788544 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 171385473 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 117285577 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚284926793644032 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 _step‚516 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12523457547264 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45544429 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚244506130347120 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12523457547264 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119990321 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 291402138 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚256990472244224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 % _timestamp‚1.7884712953281949e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12523691215872 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚269514163460096 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚272190063147520 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚284927104009216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12488182496648 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚256990454241280 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚256990454241280 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 171411817 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45545100 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119990319 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12736730496512 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12737016921600 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚256994329883128 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12487953515564 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 171411817 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚256990472244224 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _runtime‚7348.513242225 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12738725329920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12525655255040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚244543677743904 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45550977 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120008594 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚272231771057664 trainer/global_step‚6299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 171438535 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚269555413536256 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12525421573120 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12739013838848 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚284970784896512 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚257033818729746 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚269555413536256 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 _runtime‚ 7349.47214638 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚257029758281216 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚257029740278272 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 171438535 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117302927 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 291447131 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45551662 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120008594 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 291420759 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚257033572709094 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884712962867107e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 291447131 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12525655255040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚257029758281216 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚272231747117568 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12489912004550 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚284970472447488 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚257029740278272 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117302927 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 291447131 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45551662 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120008594 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 291420759 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚257033572709094 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884712962867107e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 291447131 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12525655255040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚257029758281216 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚272231747117568 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12489912004550 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚284970472447488 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚257029740278272 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12738725329920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12525655255040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚244543677743904 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45550977 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120008594 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚272231771057664 trainer/global_step‚6299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 171438535 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚269555413536256 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12525421573120 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12739013838848 KEDeó@r, viceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚284970784896512 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚257033818729746 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚269555413536256 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 _runtime‚ 7349.47214638…}_\L2Ù- _wandbruntime‚7349  _runtime‚7349¹Ôƒ_ÚÙ _step‚518 .loss/FRt4/detection_loss‚2.935967722109386 0loss/SRqqt1/detection_loss‚2.082082503117048 1loss/SRqqt4/detection_loss‚2.8585173225402833 /loss/FBt2/detection_loss‚2.5089085427197544 /loss/FBt4/detection_loss‚2.1662708599777782 0loss/FRt1/assignment_loss‚2.6337768211960793 0loss/FRt4/assignment_loss‚3.1655993163585663 2loss/SRqqt3/assignment_loss‚0.8943078926442047 /loss/FBt3/detection_loss‚0.7630059940662856 2loss/SRqqt2/assignment_loss‚1.5234170367320379 epoch‚5 1loss/FBt2/assignment_loss‚0.10460929066167327 %loss/total_loss‚43.41543801562698 trainer/global_step‚6299 _runtime‚7349.473209245 1loss/FBt1/assignment_loss‚0.39422524543273313 1loss/FBt3/assignment_loss‚0.07931622257437342 /loss/FRt2/detection_loss‚2.4225928740906966 1loss/SRqqt3/detection_loss‚0.7441846950767705 % _timestamp‚1.7884712962885234e+09 /loss/FRt2/assignment_loss‚3.798511728644371 1loss/FBt4/assignment_loss‚0.26047970762644634 /loss/FRt3/detection_loss‚1.4362963136551636 /loss/FBt1/detection_loss‚2.2581003467241922 0loss/FRt3/assignment_loss‚1.2369215020111628 2loss/SRqqt1/assignment_loss‚1.6150843049757755 2loss/SRqqt4/assignment_loss‚1.9032204073667527 /loss/FRt1/detection_loss‚2.7647359356284142 0loss/SRqqt2/detection_loss‚2.865305429697037·KïdÛÞ 0loss/FRt1/assignment_loss‚2.6337768211960793 0loss/FRt4/assignment_loss‚3.1655993163585663 2loss/SRqqt3/assignment_loss‚0.8943078926442047 /loss/FBt3/detection_loss‚0.7630059940662856 2loss/SRqqt2/assignment_loss‚1.5234170367320379 epoch‚5 1loss/FBt2/assignment_loss‚0.10460929066167327 %loss/total_loss‚43.41543801562698 trainer/global_step‚6299 _runtime‚7349.473209245 1loss/FBt1/assignment_loss‚0.39422524543273313 1loss/FBt3/assignment_loss‚0.07931622257437342 /loss/FRt2/detection_loss‚2.4225928740906966 1loss/SRqqt3/detection_loss‚0.7441846950767705 % _timestamp‚1.7884712962885234e+09 /loss/FRt2/assignment_loss‚3.798511728644371 1loss/FBt4/assignment_loss‚0.26047970762644634 /loss/FRt3/detection_loss‚1.4362963136551636 /loss/FBt1/detection_loss‚2.2581003467241922 0loss/FRt3/assignment_loss‚1.2369215020111628 2loss/SRqqt1/assignment_loss‚1.6150843049757755 2loss/SRqqt4/assignment_loss‚1.9032204073667527 /loss/FRt1/detection_loss‚2.7647359356284142 0loss/SRqqt2/detection_loss‚2.865305429697037 _step‚518 .loss/FRt4/detection_loss‚2.935967722109386 0loss/SRqqt1/detection_loss‚2.082082503117048 1loss/SRqqt4/detection_loss‚2.8585173225402833 /loss/FBt2/detection_loss‚2.5089085427197544 /loss/FBt4/detection_loss‚2.1662708599777782†y’ðÿ’Ü‚(:‡ €ÐçÔñ¬À‡ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚140.145' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.03625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0SS§ÆxÝ‚(:í €ÐçÔÝêÚˆ disk.rbd2.in‚0 disk.md0p1.in‚8.890625 disk.md0.out‚ 775.1953125 disk.sdb.in‚ 4.48828125 disk.sda1.out‚0# memory_percent‚37.1403694152832 disk.sdb2.in‚ 4.48828125 disk.sda.in‚ 4.40234375 disk.dm-0.out‚94949.26171875 disk.md0p1.out‚ 775.1953125 disk.md0.in‚8.890625 disk.sdb.out‚804.1044921875 disk.sda.out‚804.1044921875 disk.rbd2.out‚ 1.962890625 disk.sda2.out‚804.1044921875# disk.nvme1n1.out‚52631.57421875) disk./.usagePercent‚83.14784913917272 disk.rbd0.in‚0.015625 network.sent‚ 1.6077505e+07 disk.sdb2.out‚804.1044921875( proc.memory.availableMB‚ 10298.921875# proc.memory.rssMB‚ 3030.24609375) proc.memory.percent‚18.49515438079834 cpu‚25.19366649397985 disk.nvme0n1.in‚ 329776.46875 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.nvme0n1.out‚ 42317.6875 disk.sda1.in‚0 disk.sda2.in‚ 4.40234375 disk.sdb1.in‚0 disk.dm-0.in‚530976.41015625# disk.nvme1n1.in‚201199.94140625 network.recv‚ 2.494269e+06$ disk./.usageGB‚2973.769832611084 disk.rbd0.out‚ 1.962890625|CmÞ‚(:… ÐçÔñó‹‡ gpu.0.gpu‚19.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚143.872' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.968 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-fÿÂß‚(:ô ÐçÔ•À߇ disk.md0.in‚8.890625 disk.rbd0.out‚ 1.962890625 disk.dm-0.out‚ 94953.1875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb2.in‚ 4.48828125) proc.memory.percent‚18.49515438079834 disk.rbd2.in‚0 disk.rbd2.out‚ 1.962890625 disk.sda.in‚ 4.40234375 network.recv‚ 2.500491e+06 disk.sdb1.in‚0 disk.sdb1.out‚0" disk.nvme0n1.out‚ 42317.7578125( proc.memory.availableMB‚ 10299.171875 network.sent‚ 1.6173782e+07 disk.md0.out‚ 776.62109375" disk.nvme1n1.out‚ 52635.4296875 disk.md0p1.out‚ 776.62109375! disk.sdb2.out‚805.59033203125$ memory_percent‚37.13884353637695# proc.memory.rssMB‚ 3030.24609375 cpu‚25.193314506530058% disk./.usageGB‚2973.6876487731934 disk.nvme0n1.in‚ 329776.46875! disk.sda2.out‚805.59033203125 disk.md0p1.in‚8.890625 proc.cpu.threads‚17) disk./.usagePercent‚83.14555124466868 disk.sdb.in‚ 4.48828125 disk.sda.out‚805.59033203125 disk.dm-0.in‚530976.41015625 disk.rbd0.in‚0.015625# disk.nvme1n1.in‚201199.94140625 disk.sdb.out‚805.59033203125 disk.sda2.in‚ 4.40234375ˆ~|;‘à‚(:† žÐçÔËȪ‡ gpu.0.gpu‚4.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚144.821' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.20525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z*19‚á‚(:÷ žÐçÔã½Þ‡ network.sent‚ 1.6178978e+07 network.recv‚ 2.506381e+06 disk.rbd2.out‚ 1.962890625 disk.sda1.in‚0 disk.sda2.out‚807.0146484375 disk.rbd0.in‚0.015625) disk./.usagePercent‚83.15286911711702# disk.nvme1n1.in‚201199.94140625# disk.nvme0n1.out‚42317.81640625 disk.dm-0.in‚530976.41015625 disk.md0p1.out‚ 777.98046875 disk.sdb2.in‚ 4.48828125 cpu‚25.192957138782592 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.sda.out‚807.0146484375 disk.sdb2.out‚807.0146484375 disk.md0.in‚8.890625% memory_percent‚37.139081954956055 proc.cpu.threads‚17% disk./.usageGB‚2973.9493713378906" disk.nvme1n1.out‚ 52798.9609375 disk.sdb1.in‚0 disk.nvme0n1.in‚ 329776.46875 disk.sda1.out‚0 disk.sda2.in‚ 4.40234375) proc.memory.availableMB‚ 10299.1328125) proc.memory.percent‚18.49515438079834 disk.md0.out‚ 777.98046875 disk.sdb.out‚807.0146484375 disk.md0p1.in‚8.890625 disk.sdb.in‚ 4.48828125 disk.sda.in‚ 4.40234375 disk.dm-0.out‚95116.77734375 disk.rbd0.out‚ 1.962890625# proc.memory.rssMB‚ 3030.246093757’’â‚(:‡ ­ÐçÔËÒ†‡ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚136.743' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.18575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºzã‚(:ï ­ÐçÔèÕ‡ disk.rbd0.out‚ 1.962890625 disk.nvme1n1.out‚52902.5# proc.memory.rssMB‚ 3030.24609375 cpu‚25.192635351434188 disk.sda.in‚ 4.40234375 disk.dm-0.out‚ 95220.46875 disk.md0.in‚8.890625# disk.nvme1n1.in‚201199.94140625 network.recv‚ 2.517792e+06 disk.sdb1.out‚0 disk.sdb2.in‚ 4.48828125 proc.cpu.threads‚17 disk.sda.out‚808.77099609375 disk.md0p1.in‚8.890625! disk.sdb2.out‚808.77099609375 disk.rbd0.in‚0.015625* proc.memory.availableMB‚10298.87890625) proc.memory.percent‚18.49515438079834% disk./.usageGB‚2974.0743141174316 disk.dm-0.in‚530976.41015625 disk.rbd2.in‚0 disk.nvme0n1.out‚ 42317.96875 network.sent‚ 1.6190754e+07 disk.sdb.in‚ 4.48828125 disk.nvme0n1.in‚ 329776.46875 disk.sda1.out‚0 disk.sdb1.in‚0 disk.sda1.in‚0 disk.sda2.in‚ 4.40234375 disk.md0p1.out‚ 779.66796875 disk.md0.out‚ 779.66796875 disk.rbd2.out‚ 1.962890625% memory_percent‚37.140631675720215( disk./.usagePercent‚83.1563625695256 disk.sdb.out‚808.77099609375! disk.sda2.out‚808.77099609375³bó2ä- _wandbruntime‚7402  _runtime‚7402Ÿ¬´îÇåÁ % _timestamp‚1.7884713493307776e+09 _runtime‚7402.515737166 _step‚519 & lr-AdamW/pg1‚0.0004479467493868726 & lr-AdamW/pg2‚0.0004479467493868726 trainer/global_step‚6349êä=ÌæÆ & lr-AdamW/pg1‚0.0004479467493868726 & lr-AdamW/pg2‚0.0004479467493868726 trainer/global_step‚6349 % _timestamp‚1.7884713493307776e+09 _runtime‚7402.515737166 _step‚519‡Ç›¹2ç- _wandbruntime‚7402  _runtime‚7402ÖRXaî'èèO GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 trainer/global_step‚6349 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚520 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120904069 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚271576681016320 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 172749769 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚258968539774392 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72281250 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚287111179918336 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12836894328320 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 172749769 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚246382648218320 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12621673253888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚258954774093824 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚680 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12585891556072 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45888373 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚258954756090880 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72281248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚258954756090880 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚274274285590016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚271576429344768 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12621906922496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12836605806080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 % _timestamp‚1.7884713493319328e+09 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚271576429344768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 _runtime‚7402.516953951 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12621906922496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚274274261649920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12586120537156 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 293653840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚258968785794836 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120904069 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624MĈ5ó'éíO M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120904069 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚271576681016320 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 172749769 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚258968539774392 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72281250 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚287111179918336 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12836894328320 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 172749769 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚246382648218320 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12621673253888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚258954774093824 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚680 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12585891556072 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45888373 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚258954756090880 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72281248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚258954756090880 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚274274285590016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚271576429344768 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12621906922496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12836605806080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 % _timestamp‚1.7884713493319328e+09 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚271576429344768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 _runtime‚7402.516953951 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12621906922496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚274274261649920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12586120537156 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 293653840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚258968785794836 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120904069 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 trainer/global_step‚6349 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚520ˆ•ê·Õ2ê- _wandbruntime‚7403  _runtime‚7403îÝñ­ëñM C=DeviceSì~_Òæ&tatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚258994042127872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12623870961664 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 172776487 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 172776487 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚259008274641454 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚671 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 293698833 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12587850045058 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45895729 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120922344 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 trainer/global_step‚6349 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚521 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 172750117 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚669 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚246420195615104 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 172750117 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚271617931092480 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚258994060130816 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120922346 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118187110 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 293672461 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12623637279744 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12623637279744 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚271617931092480 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72292052 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚246420178575744 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12588079026350 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚258994042127872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12623870961664 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚258994060130816 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45895060 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 293672461 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚274315945619968 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚274315969560064 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884713503055654e+09 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0K†–èü&ìöM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45895060 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 293672461 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚274315945619968 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚274315969560064 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884713503055654e+09 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚258994042127872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12623870961664 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 172776487 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 172776487 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚259008274641454 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚671 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 293698833 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12587850045058 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45895729 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120922344 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 trainer/global_step‚6349 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚521 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 172750117 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚669 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚246420195615104 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 172750117 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚271617931092480 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚258994060130816 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120922346 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118187110 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 293672461 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12623637279744 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12623637279744 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚271617931092480 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72292052 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚246420178575744 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12588079026350 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚258994042127872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12623870961664 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚258994060130816 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792‰n+ÝÏ2í- _wandbruntime‚7403  _runtime‚7403^Š1ZîÔ 2loss/SRqqt4/assignment_loss‚1.7520932242274285 1loss/FBt1/assignment_loss‚0.31118288483675216 /loss/FRt3/detection_loss‚1.3274724320331706 0loss/SRqqt2/detection_loss‚2.614693852762381 trainer/global_step‚6349 /loss/FRt1/assignment_loss‚2.901440539143302 1loss/FBt4/assignment_loss‚0.15788607754583633 /loss/FRt1/detection_loss‚2.6966227022084324 &loss/total_loss‚41.523107952934815 epoch‚5 2loss/SRqqt3/assignment_loss‚0.9805446414516197 .loss/FRt2/detection_loss‚2.152693375015811 /loss/FRt4/detection_loss‚3.1978817601357736 0loss/SRqqt3/detection_loss‚0.916475644557301 -loss/FBt1/detection_loss‚1.85715837445524 2loss/SRqqt2/assignment_loss‚1.4943276966611545 1loss/FBt3/assignment_loss‚0.02229337945919989 .loss/FBt3/detection_loss‚0.558372103572382 0loss/FRt4/assignment_loss‚2.9563979737220274 0loss/SRqqt1/assignment_loss‚1.77654406705568 1loss/SRqqt4/detection_loss‚2.6635760088761646 /loss/FBt2/detection_loss‚2.6156033664941787 1loss/SRqqt1/detection_loss‚1.9691528747248095 % _timestamp‚1.7884713503090045e+09 _step‚522 /loss/FBt4/detection_loss‚1.8748610185252295 /loss/FRt2/assignment_loss‚3.201856117499502 0loss/FRt3/assignment_loss‚1.2978642107565193 1loss/FBt2/assignment_loss‚0.22611362721492015 _runtime‚7403.493740359÷OF_ïÙ 1loss/SRqqt1/detection_loss‚1.9691528747248095 % _timestamp‚1.7884713503090045e+09 _step‚522 /loss/FBt4/detection_loss‚1.8748610185252295 /loss/FRt2/assignment_loss‚3.201856117499502 0loss/FRt3/assignment_loss‚1.2978642107565193 1loss/FBt2/assignment_loss‚0.22611362721492015 _runtime‚7403.493740359 2loss/SRqqt4/assignment_loss‚1.7520932242274285 1loss/FBt1/assignment_loss‚0.31118288483675216 /loss/FRt3/detection_loss‚1.3274724320331706 0loss/SRqqt2/detection_loss‚2.614693852762381 trainer/global_step‚6349 /loss/FRt1/assignment_loss‚2.901440539143302 1loss/FBt4/assignment_loss‚0.15788607754583633 /loss/FRt1/detection_loss‚2.6966227022084324 &loss/total_loss‚41.523107952934815 epoch‚5 2loss/SRqqt3/assignment_loss‚0.9805446414516197 .loss/FRt2/detection_loss‚2.152693375015811 /loss/FRt4/detection_loss‚3.1978817601357736 0loss/SRqqt3/detection_loss‚0.916475644557301 -loss/FBt1/detection_loss‚1.85715837445524 2loss/SRqqt2/assignment_loss‚1.4943276966611545 1loss/FBt3/assignment_loss‚0.02229337945919989 .loss/FBt3/detection_loss‚0.558372103572382 0loss/FRt4/assignment_loss‚2.9563979737220274 0loss/SRqqt1/assignment_loss‚1.77654406705568 1loss/SRqqt4/detection_loss‚2.6635760088761646 /loss/FBt2/detection_loss‚2.6156033664941787Šná‘’ð‚(:‡ ¼ÐçÔ·Òþ‡ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚139.251' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.81275 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öF1Ðñ‚(:ô ¼ÐçԌӖˆ disk.rbd0.out‚ 1.962890625% disk./.usageGB‚2974.3304023742676 disk.dm-0.out‚95368.19140625 disk.sdb2.in‚ 4.48828125# disk.nvme1n1.in‚201199.94140625 disk.sdb1.in‚0 disk.sda.in‚ 4.40234375 disk.sda.out‚810.5224609375 disk.md0.out‚ 781.3671875 disk.sdb.in‚ 4.48828125 disk.sdb1.out‚0 network.recv‚ 2.522099e+06 disk.nvme0n1.in‚ 329776.46875" disk.nvme1n1.out‚ 53050.1640625 proc.cpu.threads‚17 disk.sda2.in‚ 4.40234375 disk.md0p1.out‚ 781.3671875% memory_percent‚37.139296531677246* proc.memory.availableMB‚10299.09765625 disk.dm-0.in‚530976.41015625 disk.sda2.out‚810.5224609375 disk.sdb2.out‚810.5224609375 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 1.962890625# proc.memory.rssMB‚ 3030.24609375) proc.memory.percent‚18.49515438079834 disk.rbd2.in‚0 disk.sdb.out‚810.5224609375 network.sent‚ 1.6207466e+07' disk./.usagePercent‚83.163522904368# disk.nvme0n1.out‚42318.02734375 disk.md0p1.in‚8.890625 disk.md0.in‚8.890625 cpu‚25.192243518551294 disk.sda1.in‚0 disk.sda1.out‚0Àg€Uxò‚(:í ËÐçÔ÷Ë–‡ disk.md0p1.in‚8.890625" disk.nvme0n1.out‚ 42318.3984375 cpu‚25.191897494596276 disk.dm-0.in‚530976.41015625 disk.md0.in‚8.890625 disk.rbd2.in‚0# disk.nvme1n1.out‚53153.78515625* proc.memory.availableMB‚10299.34765625) proc.memory.percent‚18.49515438079834 network.recv‚ 2.528781e+06 disk.sdb.out‚812.0771484375 disk.sdb1.out‚0 disk.sda.out‚812.0771484375 disk.nvme0n1.in‚ 329776.46875 disk.sda2.in‚ 4.40234375 disk.sda2.out‚812.0771484375 disk.sdb2.out‚812.0771484375( disk./.usagePercent‚83.1634960259139 disk.sda.in‚ 4.40234375 disk.dm-0.out‚95472.18359375 disk.md0p1.out‚782.875 disk.md0.out‚782.875 disk.rbd0.out‚ 1.962890625 disk.rbd2.out‚ 1.962890625% memory_percent‚37.137770652770996 disk.sdb.in‚ 4.48828125 disk.sda1.in‚0 disk.sdb2.in‚ 4.48828125 disk.rbd0.in‚0.015625# disk.nvme1n1.in‚201199.94140625# proc.memory.rssMB‚ 3030.24609375 disk.sdb1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 network.sent‚ 1.6303845e+07% disk./.usageGB‚2974.3294410705566kòqt’ó‚(:‡ ËÐçÔ°çÇ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚129.005' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.25125 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$î\µ‘ô‚(:† ÚÐçÔ¦¥Å‡ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.966' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.9915 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Bg›†õ‚(:û ÚÐçÔƒÑì‡ disk.sdb1.out‚0 disk.md0.in‚8.890625 network.recv‚ 2.547211e+06* proc.memory.availableMB‚10298.95703125) proc.memory.percent‚18.49515438079834 proc.cpu.threads‚17) disk./.usagePercent‚83.17049722324236 disk.sdb2.in‚ 4.48828125 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 1.962890625 disk.sdb.in‚ 4.48828125 disk.dm-0.out‚95611.87109375# disk.nvme0n1.out‚42318.45703125 network.sent‚ 1.6314943e+07% disk./.usageGB‚2974.5798377990723 disk.md0p1.out‚ 784.38671875 disk.rbd2.in‚0 disk.nvme0n1.in‚ 329776.46875 disk.dm-0.in‚530976.41015625 disk.sda.out‚813.64501953125 disk.sda1.out‚0 disk.sda2.in‚ 4.40234375! disk.sda2.out‚813.64501953125 disk.rbd0.out‚ 1.962890625# disk.nvme1n1.in‚201199.94140625 disk.sdb1.in‚0 cpu‚25.191541041624305 disk.sda1.in‚0 disk.md0p1.in‚8.890625" disk.nvme1n1.out‚ 53293.4140625 disk.sda.in‚ 4.40234375$ memory_percent‚37.14015483856201! disk.sdb2.out‚813.64501953125 disk.sdb.out‚813.64501953125 disk.md0.out‚ 784.38671875# proc.memory.rssMB‚ 3030.24609375)­h32ö- _wandbruntime‚7454  _runtime‚7454W$†;É÷à % _timestamp‚1.7884714016461518e+09 _runtime‚7454.831026189 _step‚523 ' lr-AdamW/pg1‚0.00044201466008804934 ' lr-AdamW/pg2‚0.00044201466008804934 trainer/global_step‚63999ÇÄÎøÈ ' lr-AdamW/pg1‚0.00044201466008804934 ' lr-AdamW/pg2‚0.00044201466008804934 trainer/global_step‚6399 % _timestamp‚1.7884714016461518e+09 _runtime‚7454.831026189 _step‚523‹Ìåˆ2ù- _wandbruntime‚7454  _runtime‚7454þ;‹–úèO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121817821 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 174087721 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 174087721 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚248259183128880 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 295905542 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 119058379 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 174061377 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 trainer/global_step‚6399 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚248259166089520 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12684058577664 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 _runtime‚7454.832059953 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121817821 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12720122629120 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚260942995686100 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batc…Bâ Xh_start/active.all.freed‚ 295879196 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚276358484092416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72821348 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚273639198572544 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121817819 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚654 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 % _timestamp‚1.7884714016468697e+09 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 119057723 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12936728941056 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46237029 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚289295213033472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚260919075943424 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚273638946900992 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12719888960512 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12936440418816 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚260943241706544 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚260919057940480 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46236375 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚260919075943424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚273639198572544 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚273638946900992 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2€Lçqó'ûíO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚260919075943424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚273639198572544 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚273638946900992 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121817821 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 174087721 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 174087721 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚248259183128880 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 295905542 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 119058379 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 174061377 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 trainer/global_step‚6399 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚248259166089520 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12684058577664 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 _runtime‚7454.832059953 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121817821 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12720122629120 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚260942995686100 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 295879196 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚276358484092416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72821348 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚273639198572544 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121817819 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚654 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 % _timestamp‚1.7884714016468697e+09 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 119057723 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12936728941056 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46237029 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚289295213033472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚260919075943424 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚273638946900992 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12719888960512 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12936440418816 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚260943241706544 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚260919057940480 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46236375 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0Œ5pn’ü‚(:‡ éÐçÔô†‰ gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚143.595' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.89875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á)cN€ý‚(:õ éÐçÔÉ–Þ‰ disk.sda.in‚ 4.40234375 disk.sdb2.in‚ 4.48828125 disk.rbd2.out‚ 1.962890625" disk.nvme0n1.out‚ 42318.6015625) disk./.usagePercent‚83.16275804169216$ disk./.usageGB‚2974.303047180176# disk.nvme1n1.in‚201199.94140625 disk.sdb.in‚ 4.48828125 disk.sda2.in‚ 4.40234375 disk.rbd2.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0.015625$ memory_percent‚37.14404106140137 proc.cpu.threads‚17! disk.nvme1n1.out‚ 53529.890625 disk.sda.out‚815.63916015625 disk.dm-0.out‚ 95848.4921875 disk.nvme0n1.in‚ 329776.46875! disk.sda2.out‚815.63916015625! disk.sdb2.out‚815.63916015625 disk.md0.in‚8.890625 disk.dm-0.in‚530976.41015625 disk.sda1.in‚0 disk.md0p1.out‚ 786.31640625 disk.md0.out‚ 786.31640625# proc.memory.rssMB‚ 3030.24609375 disk.sdb.out‚815.63916015625 disk.sdb1.in‚0 disk.rbd0.out‚ 1.962890625) proc.memory.availableMB‚ 10298.3203125 cpu‚25.19121128403019 network.sent‚ 1.6324126e+07 network.recv‚ 2.558077e+06 disk.md0p1.in‚8.890625) proc.memory.percent‚18.49515438079834 disk.sdb1.out‚0çËï€2þ- _wandbruntime‚7455  _runtime‚7455#ÆV³ø&ÿòM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚276400168062464 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚260982484532510 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚703 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12685788085566 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚260958361980416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119075221 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚289338579331072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚260982730553162 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚248296696446944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚260958343977472 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12686017066858 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12721852986368 trainer/global_step‚6399 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 174114439 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚273680448648704 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46243774 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚289338889682944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 295950535 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 295950535 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _runtime‚7455.813783742 _step‚525 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚705 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12722086668288 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121836094 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 % _timestamp‚1.7884714026273973e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚276400144122368 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121836096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121836094 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46243071 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12938721620480 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 295924163 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 295924163 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72832152 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 174114439 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12938435208704 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚72832150 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121836096 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 174088069 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121836096 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 174088069 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚703 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12685788085566 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚260958361980416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119075221 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚289338579331072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚260982730553162 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚248296696446944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚260958343977472 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12686017066858 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12721852986368 trainer/global_step‚6399 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 174114439 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚273680448648704 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46243774 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚289338889682944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 295950535 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 295950535 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _runtime‚7455.813783742 _step‚525 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚705 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12722086668288 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121836094 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 % _timestamp‚1.7884714026273973e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚276400144122368 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121836096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121836094 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46243071 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12938721620480 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 295924163 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582—Km2- _wandbruntime‚7455  _runtime‚7455­­\†a‚Û /loss/FRt1/detection_loss‚2.5146403119844547 0loss/FRt3/assignment_loss‚1.4614959612284621 1loss/SRqqt2/assignment_loss‚1.815729613105456 /loss/FBt2/detection_loss‚3.4050542371613637 _runtime‚7455.814813006 3loss/FBt2/assignment_loss‚0.0008889270491993427 1loss/SRqqt1/detection_loss‚1.8209661005473718 0loss/SRqqt4/detection_loss‚2.416588069871068 epoch‚5 2loss/SRqqt3/assignment_loss‚0.9360436833956662 2loss/FBt1/assignment_loss‚0.010540141066693801 2loss/FBt3/assignment_loss‚0.022192986044037213 /loss/FRt3/detection_loss‚1.4335078110739792 1loss/SRqqt3/detection_loss‚1.0581525966525078 /loss/FRt1/assignment_loss‚2.993460069684421 /loss/FBt4/detection_loss‚2.5566089351971946 /loss/FRt2/detection_loss‚2.1014805403961376 0loss/FRt2/assignment_loss‚3.3899568102576514 /loss/FRt4/assignment_loss‚3.279637801647186 /loss/FRt4/detection_loss‚3.4522944182157516 1loss/SRqqt2/detection_loss‚2.4451055548809193 _step‚526 1loss/FBt4/assignment_loss‚0.05512994900316749 2loss/SRqqt4/assignment_loss‚1.5297486411873251 /loss/FBt3/detection_loss‚0.5722524138001649 2loss/SRqqt1/assignment_loss‚1.7757361568964836 .loss/FBt1/detection_loss‚2.384680857261022 &loss/total_loss‚43.431892587607685 trainer/global_step‚6399 # _timestamp‚1.78847140263012e+09”‘>…fƒà trainer/global_step‚6399 # _timestamp‚1.78847140263012e+09 /loss/FRt1/detection_loss‚2.5146403119844547 0loss/FRt3/assignment_loss‚1.4614959612284621 1loss/SRqqt2/assignment_loss‚1.815729613105456 /loss/FBt2/detection_loss‚3.4050542371613637 _runtime‚7455.814813006 3loss/FBt2/assignment_loss‚0.0008889270491993427 1loss/SRqqt1/detection_loss‚1.8209661005473718 0loss/SRqqt4/detection_loss‚2.416588069871068 epoch‚5 2loss/SRqqt3/assignment_loss‚0.9360436833956662 2loss/FBt1/assignment_loss‚0.010540141066693801 2loss/FBt3/assignment_loss‚0.022192986044037213 /loss/FRt3/detection_loss‚1.4335078110739792 1loss/SRqqt3/detection_loss‚1.0581525966525078 /loss/FRt1/assignment_loss‚2.993460069684421 /loss/FBt4/detection_loss‚2.5566089351971946 /loss/FRt2/detection_loss‚2.1014805403961376 0loss/FRt2/assignment_loss‚3.3899568102576514 /loss/FRt4/assignment_loss‚3.279637801647186 /loss/FRt4/detection_loss‚3.4522944182157516 1loss/SRqqt2/detection_loss‚2.4451055548809193 _step‚526 1loss/FBt4/assignment_loss‚0.05512994900316749 2loss/SRqqt4/assignment_loss‚1.5297486411873251 /loss/FBt3/detection_loss‚0.5722524138001649 2loss/SRqqt1/assignment_loss‚1.7757361568964836 .loss/FBt1/detection_loss‚2.384680857261022 &loss/total_loss‚43.431892587607685Žêêh’„‚(:‡ øÐçÔá«ä‡ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚139.251' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.81275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÛÅ¥¶}…‚(:ò øÐçÔô±Úˆ# proc.memory.rssMB‚ 3030.24609375) proc.memory.percent‚18.49515438079834 disk.rbd2.out‚ 1.962890625 cpu‚25.190935861700495 disk.md0.in‚8.890625 disk.rbd2.in‚0 disk.nvme1n1.out‚ 53713.59375 disk.sdb.in‚ 4.48828125! disk.sda2.out‚817.19091796875* proc.memory.availableMB‚10298.76953125 disk.sda.in‚ 4.40234375" disk.nvme0n1.out‚ 42318.6640625 disk.dm-0.in‚530976.41015625 network.sent‚ 1.6420954e+07 disk.rbd0.out‚ 1.962890625 disk.sda1.in‚0# memory_percent‚37.1412992477417( disk./.usagePercent‚83.1627219904323! disk.sdb2.out‚817.19091796875 disk.dm-0.out‚ 96032.2578125 proc.cpu.threads‚17" disk./.usageGB‚2974.3017578125 disk.sdb1.out‚0 disk.md0p1.in‚8.890625 disk.sda.out‚817.19091796875 disk.sda1.out‚0# disk.nvme1n1.in‚201199.94140625 network.recv‚ 2.565693e+06 disk.sdb2.in‚ 4.48828125 disk.md0.out‚ 787.79296875 disk.rbd0.in‚0.015625 disk.sdb.out‚817.19091796875 disk.nvme0n1.in‚ 329776.46875 disk.sda2.in‚ 4.40234375 disk.sdb1.in‚0 disk.md0p1.out‚ 787.79296875¢sìP‘†‚(:† ‡ÑçÔ°²Œ† gpu.0.gpu‚24.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚124.994' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.2485 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07ùòTu‡‚(:ê ‡ÑçÔËü›‡ disk.sdb2.in‚ 4.48828125 disk.sdb2.out‚818.5458984375 disk.sda2.in‚ 4.40234375 disk.sda2.out‚818.5458984375# disk.nvme1n1.in‚201199.94140625 disk.nvme1n1.out‚ 54044.46875 disk.nvme0n1.in‚ 329776.46875 network.sent‚ 1.6517465e+07 disk.md0p1.in‚8.890625 disk.md0.out‚ 789.08984375 disk.sdb1.out‚0 disk.nvme0n1.out‚ 42319.28125# proc.memory.rssMB‚ 3030.24609375 cpu‚25.19063177725745 disk.rbd0.in‚0.015625 disk.dm-0.out‚96363.75 network.recv‚ 2.571649e+06$ disk./.usageGB‚2974.827251434326 disk.md0.in‚8.890625 disk.rbd2.in‚0 disk.sdb.in‚ 4.48828125 disk.sda.in‚ 4.40234375 disk.sda1.out‚0) proc.memory.percent‚18.49515438079834 disk.md0p1.out‚ 789.08984375 disk.rbd2.out‚ 1.962890625 disk.sdb.out‚818.5458984375 disk.sda.out‚818.5458984375 disk.sdb1.in‚0 disk.sda1.in‚0* proc.memory.availableMB‚10299.01953125 proc.cpu.threads‚17) disk./.usagePercent‚83.17741501203471 disk.rbd0.out‚ 1.962890625 disk.dm-0.in‚530976.41015625$ memory_percent‚37.13977336883545þö=yœˆ‚(:‘ –ÑçÔ¡Û㇠gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0YQ–Ö~‰‚(:ó –ÑçÔŠšˆ disk.sdb.in‚ 4.48828125 disk.sda.out‚ 820.326171875 disk.sdb2.out‚ 820.326171875% disk./.usageGB‚2974.8293533325195 disk.dm-0.out‚96496.87890625 disk.rbd2.out‚ 1.962890625* proc.memory.availableMB‚10298.51953125 cpu‚25.190311664655624 disk.sdb1.out‚0 disk.sda1.in‚0 disk.rbd0.out‚ 1.962890625# proc.memory.rssMB‚ 3030.24609375) proc.memory.percent‚18.49515438079834# disk.nvme0n1.out‚42319.33984375 disk.md0p1.in‚8.890625 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 network.recv‚ 2.583101e+06) disk./.usagePercent‚83.17747378198791 disk.sdb.out‚ 820.326171875 disk.sdb1.in‚0 disk.md0.in‚8.890625" disk.nvme1n1.out‚ 54177.5390625 disk.sda.in‚ 4.40234375 disk.nvme0n1.in‚ 329776.46875 disk.md0.out‚ 790.80859375 network.sent‚ 1.6527193e+07 disk.sda1.out‚0 disk.md0p1.out‚ 790.80859375 disk.sdb2.in‚ 4.48828125$ memory_percent‚37.14282512664795 proc.cpu.threads‚17 disk.dm-0.in‚530976.41015625 disk.sda2.in‚ 4.40234375 disk.sda2.out‚ 820.326171875# disk.nvme1n1.in‚201199.94140625zVMÅ2Š- _wandbruntime‚7508  _runtime‚7508à«ŸÅ‹¿ % lr-AdamW/pg1‚0.000436076362950275 % lr-AdamW/pg2‚0.000436076362950275 trainer/global_step‚6449 % _timestamp‚1.7884714556950107e+09 _runtime‚7508.879998168 _step‚5272\Ž‚ÊŒÄ trainer/global_step‚6449 % _timestamp‚1.7884714556950107e+09 _runtime‚7508.879998168 _step‚527 % lr-AdamW/pg1‚0.000436076362950275 % lr-AdamW/pg2‚0.000436076362950275—'ß2- _wandbruntime‚7508  _runtime‚7508Š˜m3î'ŽèO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12818338335744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122731569 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12818338335744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46586321 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 122731571 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚262917697618252 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 175425673 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 175399329 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚291478831124992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 119947089 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13036460992512 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚275701464457216 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 298157244 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 trainer/global_step‚6449 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚250135701000080 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73361450 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _runtime‚7508.881575425 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚262883359790080 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚278442658654720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚262917451597808 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚262883359790080 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12818104667136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚275701464457216 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 % _timestamp‚1.7884714556961446e+09 _step‚528 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚291479143587328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 175425673 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 298130898 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175399329 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚275701716128768 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122731569 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122731571 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13036172470272 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚275701716128768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚250135683960720 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12781996618172 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12781767637088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚73361448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46585641 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608¾÷¥Ò&íO G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 175399329 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚291478831124992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 119947089 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13036460992512 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚275701464457216 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 298157244 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 trainer/global_step‚6449 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚250135701000080 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73361450 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _runtime‚7508.881575425 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚262883359790080 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚278442658654720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚262917451597808 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚262883359790080 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12818104667136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚275701464457216 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 % _timestamp‚1.7884714556961446e+09 _step‚528 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚291479143587328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 175425673 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 298130898 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175399329 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚275701716128768 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122731569 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122731571 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13036172470272 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚275701716128768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚250135683960720 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12781996618172 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12781767637088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚73361448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46585641 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12818338335744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122731569 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12818338335744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46586321 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 122731571 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚262917697618252 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 175425673 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119965151 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚262922645827072 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73372250 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13038166061568 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12820302374912 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚6449 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚262922645827072 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚291522825329664 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 175452391 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚262957186464870 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12820302374912 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13038458764800 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46592901 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚7509.971071903 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 298202237 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 175426021 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚262956940444218 % _timestamp‚1.7884714567848637e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 298175865 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 119965831 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46593579 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚250173214318144 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12783955107366 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 175452391 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 298202237 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12820068692992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12820068692992 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122749844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 175426021 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736ÿ,]!ý&’÷M H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 298202237 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12820068692992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12820068692992 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122749844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 175426021 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 298175865 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119965151 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚262922645827072 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73372250 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13038166061568 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12820302374912 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚6449 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚262922645827072 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚291522825329664 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 175452391 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚262957186464870 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12820302374912 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13038458764800 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46592901 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚7509.971071903 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 298202237 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 175426021 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚262956940444218 % _timestamp‚1.7884714567848637e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 298175865 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 119965831 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46593579 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚250173214318144 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12783955107366 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 175452391 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582‘ÚÃ-a2“- _wandbruntime‚7509  _runtime‚7509 ¬(ê`”Ú 0loss/FRt1/assignment_loss‚2.8978044589360556 2loss/SRqqt1/assignment_loss‚1.5461422299157899 1loss/SRqqt4/assignment_loss‚1.377996768305699 1loss/SRqqt3/detection_loss‚0.7815104579973605 1loss/SRqqt4/detection_loss‚2.4482685069243115 /loss/FBt3/detection_loss‚0.7228861940970671 trainer/global_step‚6449 1loss/FBt4/assignment_loss‚0.06410315372752962 .loss/FRt2/detection_loss‚2.470637353517845 % _timestamp‚1.7884714567874663e+09 _step‚530 1loss/FBt2/assignment_loss‚0.04153824284582119 3loss/FBt3/assignment_loss‚0.0044157542679287235 /loss/FRt3/detection_loss‚1.4845046169695355 0loss/SRqqt2/detection_loss‚2.960539514603822 %loss/total_loss‚42.64753235406669 _runtime‚7509.972302038 2loss/SRqqt3/assignment_loss‚0.9989169859878158 .loss/FRt4/detection_loss‚3.554188440827762 /loss/FBt2/detection_loss‚2.7052865353497593 epoch‚5 /loss/FRt1/detection_loss‚2.9951486428578695 /loss/FRt2/assignment_loss‚3.619646033725223 /loss/FBt1/detection_loss‚2.5406056859276513 0loss/FRt4/assignment_loss‚2.2437057004255405 1loss/FBt1/assignment_loss‚0.08682948541129255 0loss/FRt3/assignment_loss‚1.1834740350323338 2loss/SRqqt2/assignment_loss‚1.7475495701250823 1loss/SRqqt1/detection_loss‚1.7896605921241473 /loss/FBt4/detection_loss‚2.3821733941634498Wü{e•ß 1loss/FBt4/assignment_loss‚0.06410315372752962 .loss/FRt2/detection_loss‚2.470637353517845 % _timestamp‚1.7884714567874663e+09 _step‚530 1loss/FBt2/assignment_loss‚0.04153824284582119 3loss/FBt3/assignment_loss‚0.0044157542679287235 /loss/FRt3/detection_loss‚1.4845046169695355 0loss/SRqqt2/detection_loss‚2.960539514603822 %loss/total_loss‚42.64753235406669 _runtime‚7509.972302038 2loss/SRqqt3/assignment_loss‚0.9989169859878158 .loss/FRt4/detection_loss‚3.554188440827762 /loss/FBt2/detection_loss‚2.7052865353497593 epoch‚5 /loss/FRt1/detection_loss‚2.9951486428578695 /loss/FRt2/assignment_loss‚3.619646033725223 /loss/FBt1/detection_loss‚2.5406056859276513 0loss/FRt4/assignment_loss‚2.2437057004255405 1loss/FBt1/assignment_loss‚0.08682948541129255 0loss/FRt3/assignment_loss‚1.1834740350323338 2loss/SRqqt2/assignment_loss‚1.7475495701250823 1loss/SRqqt1/detection_loss‚1.7896605921241473 /loss/FBt4/detection_loss‚2.3821733941634498 0loss/FRt1/assignment_loss‚2.8978044589360556 2loss/SRqqt1/assignment_loss‚1.5461422299157899 1loss/SRqqt4/assignment_loss‚1.377996768305699 1loss/SRqqt3/detection_loss‚0.7815104579973605 1loss/SRqqt4/detection_loss‚2.4482685069243115 /loss/FBt3/detection_loss‚0.7228861940970671 trainer/global_step‚6449’u–|Þ‘–‚(:† ¥ÑçÔø‡¯‡ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚136.134' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.0335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$d-ºz—‚(:ï ¥ÑçÔ”Öé‡$ disk./.usageGB‚2974.954433441162 disk.sda2.in‚ 4.40234375 disk.md0p1.out‚ 792.21875 disk.md0.out‚ 792.21875 network.recv‚ 2.58754e+06 disk.dm-0.in‚530976.41015625 disk.dm-0.out‚96742.82421875 disk.sdb2.in‚ 4.48828125! disk.sdb2.out‚821.81201171875 disk.md0.in‚8.890625 disk.rbd2.in‚0( proc.memory.availableMB‚ 10297.984375 disk.md0p1.in‚8.890625 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 1.962890625 disk.sdb.in‚ 4.48828125 disk.sdb1.out‚0) proc.memory.percent‚18.49515438079834 disk.sdb.out‚821.81201171875 disk.sda.out‚821.81201171875$ memory_percent‚37.14609146118164! disk.sda2.out‚821.81201171875 disk.nvme0n1.in‚ 329776.46875# disk.nvme1n1.in‚201199.94140625! disk.nvme1n1.out‚ 54423.328125 disk.sda.in‚ 4.40234375 disk.rbd2.out‚ 1.962890625# disk.nvme0n1.out‚42319.49609375 cpu‚25.18996933453988 network.sent‚ 1.6533048e+07) disk./.usagePercent‚83.18097107417566 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb1.in‚0# proc.memory.rssMB‚ 3030.24609375 proc.cpu.threads‚17}g)’˜‚(:‡ ´ÑçÔø¿á‡ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚139.251' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.81275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QrÓy™‚(:î ´ÑçÔüÞ–ˆ# proc.memory.rssMB‚ 3030.24609375 network.sent‚ 1.6643188e+07 disk.rbd0.in‚0.015625 disk.sda.out‚823.3193359375 disk.sda1.in‚0 disk.sdb2.in‚ 4.48828125 cpu‚25.189656145248122 disk.nvme0n1.in‚ 329776.46875 disk.md0p1.out‚ 793.671875$ memory_percent‚37.14244365692139 proc.cpu.threads‚17$ disk./.usageGB‚2975.079433441162 disk.md0.in‚8.890625 disk.md0.out‚ 793.671875" disk.nvme1n1.out‚ 54601.3515625 disk.dm-0.in‚530976.41015625 disk.sda2.out‚823.3193359375 disk.rbd0.out‚ 1.962890625 disk.sda.in‚ 4.40234375" disk.nvme0n1.out‚ 42319.5546875 disk.dm-0.out‚ 96920.90625) proc.memory.percent‚18.49515438079834) disk./.usagePercent‚83.18446612649223 disk.rbd2.out‚ 1.962890625# disk.nvme1n1.in‚201199.94140625 disk.sdb.out‚823.3193359375 disk.sdb1.in‚0 disk.sdb2.out‚823.3193359375* proc.memory.availableMB‚10298.58203125 network.recv‚ 2.594446e+06 disk.rbd2.in‚0 disk.sdb.in‚ 4.48828125 disk.sdb1.out‚0 disk.sda1.out‚0 disk.sda2.in‚ 4.40234375 disk.md0p1.in‚8.890625^ÙÐç’š‚(:‡ ÃÑçԾѸ† gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚139.251' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.81275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 DÀã„›‚(:ù ÃÑçÔÕŒ„‡ disk.sdb.in‚ 4.48828125 disk.dm-0.out‚97224.03515625 disk.sda1.out‚0* proc.memory.availableMB‚10298.47265625 network.recv‚ 2.597491e+06 disk.md0.out‚ 795.76953125 disk.sdb.out‚825.48095703125! disk.sda2.out‚825.48095703125) proc.memory.percent‚18.49515438079834 disk.md0p1.in‚8.890625! disk.sdb2.out‚825.48095703125 disk.md0.in‚8.890625# disk.nvme1n1.in‚201199.94140625 disk.sda.in‚ 4.40234375! disk.nvme0n1.out‚ 42319.734375 cpu‚25.189354711469857 disk.md0p1.out‚ 795.76953125 disk.sdb2.in‚ 4.48828125 disk.sdb1.out‚0 disk.dm-0.in‚530976.41015625 disk.sda2.in‚ 4.40234375 network.sent‚ 1.6645391e+07 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 1.962890625# disk.nvme1n1.out‚54904.30078125 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.rbd2.out‚ 1.962890625 disk.sda.out‚825.48095703125 disk.sda1.in‚0) disk./.usagePercent‚83.19148630939542 disk.sdb1.in‚0$ memory_percent‚37.14311122894287# proc.memory.rssMB‚ 3030.24609375$ disk./.usageGB‚2975.330509185791 disk.nvme0n1.in‚ 329776.46875*˜Nï‘œ‚(:† ÒÑçÔÊø̇ gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚140.478' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.1195 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýƒõ‚(:ö ÒÑçԚІˆ cpu‚25.18903942841915 disk.nvme0n1.in‚ 329776.46875 disk.dm-0.out‚ 97505.28125 disk.sda2.in‚ 4.40234375 disk.rbd0.out‚ 1.962890625# disk.nvme1n1.in‚201199.94140625) proc.memory.percent‚18.49515438079834" disk.nvme1n1.out‚ 55185.3828125 disk.rbd2.out‚ 1.962890625 network.recv‚ 2.602326e+06% disk./.usageGB‚2974.2722549438477 disk.sdb1.out‚0 disk.sdb2.in‚ 4.48828125$ memory_percent‚37.14311122894287) disk./.usagePercent‚83.16189707787687 disk.sda.in‚ 4.40234375 disk.md0.out‚ 797.54296875 disk.rbd2.in‚0 disk.sdb.in‚ 4.48828125* proc.memory.availableMB‚10298.47265625! disk.sda2.out‚827.31787109375# proc.memory.rssMB‚ 3030.24609375 proc.cpu.threads‚17 disk.sda1.out‚0 disk.md0.in‚8.890625 network.sent‚ 1.6654056e+07 disk.sda.out‚827.31787109375" disk.nvme0n1.out‚ 42319.8984375 disk.md0p1.out‚ 797.54296875 disk.sdb1.in‚0 disk.dm-0.in‚530976.41015625 disk.sda1.in‚0 disk.md0p1.in‚8.890625! disk.sdb2.out‚827.31787109375 disk.rbd0.in‚0.015625 disk.sdb.out‚827.31787109375èצ2ž- _wandbruntime‚7562  _runtime‚7562:Ó†PÆŸÀ $ _timestamp‚1.788471509814192e+09 _runtime‚7562.999086546 _step‚531 & lr-AdamW/pg1‚0.0004301330955057723 & lr-AdamW/pg2‚0.0004301330955057723 trainer/global_step‚6499œzªºË Å & lr-AdamW/pg1‚0.0004301330955057723 & lr-AdamW/pg2‚0.0004301330955057723 trainer/global_step‚6499 $ _timestamp‚1.788471509814192e+09 _runtime‚7562.999086546 _step‚531“úÁ2¡- _wandbruntime‚7563  _runtime‚7563‰d]1 ¢èO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚264847661639680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 176737281 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73901550 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚73901548 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚277763982013440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚277764233684992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚649 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚264847679642624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46932730 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275939328 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚264891907509516 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚252012218871280 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 123645319 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12916554042368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 176763625 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 trainer/global_step‚6499 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 300382600 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 176763625 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚293662894605312 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13136313387520 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚264892153529960 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _runtime‚7563.000164261 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 120834278 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12879705677596 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46933377 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299879424 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚277763982013440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚277764233684992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚649 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚264847679642624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46932730 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275939328 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚264891907509516 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚252012218871280 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 123645319 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12916554042368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 176763625 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 trainer/global_step‚6499 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 300382600 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 176763625 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚293662894605312 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13136313387520 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚264892153529960 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _runtime‚7563.000164261 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 120834278 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12879705677596 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273”‡‹Q2¤- _wandbruntime‚7563  _runtime‚75631}÷&¥ñM \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚293706884323328 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚252049749228704 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚661 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 $ _timestamp‚1.788471510760385e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12918284399616 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13138026552832 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123663594 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120852153 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚659 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12881893147874 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 300453939 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 300427567 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚280568541127168 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _step‚533 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚277805483761152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46940462 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 300427567 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 176790343 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 176763973 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚264886965679616 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚293706567680000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚280568565067264 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚252049732189344 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12881664166582 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46939803 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚277805232076288 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73912350 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 176763973 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300453939 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123663596 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚264931642376578 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚264931396355926 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123663596 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚264886947676672 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 176790343 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120852814 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12918518081536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73912352 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13138319256064 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚264886947676672 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12918284399616 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _runtime‚7563.945901504 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249Qí×ÙÕ¦öM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚264886947676672 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12918284399616 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _runtime‚7563.945901504 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚293706884323328 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚252049749228704 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚661 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 $ _timestamp‚1.788471510760385e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12918284399616 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13138026552832 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123663594 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120852153 KDDeviceStatsMonitor.on_train_batch_end/inac)úÇ­'tive_split.large_pool.peak‚36 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚659 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12881893147874 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 300453939 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 300427567 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚280568541127168 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _step‚533 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚277805483761152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46940462 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 300427567 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 176790343 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 176763973 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚264886965679616 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚293706567680000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚280568565067264 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚252049732189344 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12881664166582 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46939803 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚277805232076288 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73912350 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 176763973 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300453939 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123663596 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚264931642376578 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚264931396355926 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123663596 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚264886947676672 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 176790343 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120852814 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12918518081536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73912352 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13138319256064 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582•óüÈT2§- _wandbruntime‚7563  _runtime‚7563öPíÄ]¨× 0loss/FRt3/assignment_loss‚1.2748859777288921 1loss/FBt4/assignment_loss‚0.37616956494743475 0loss/SRqqt1/detection_loss‚1.823259257353269 0loss/SRqqt2/detection_loss‚2.331010630837193 /loss/FBt1/detection_loss‚1.8175299485524496 1loss/FBt3/assignment_loss‚0.07785017136400067 epoch‚5 % _timestamp‚1.7884715107627172e+09 /loss/FBt4/detection_loss‚2.3795459449291227 /loss/FRt4/assignment_loss‚3.379587987065315 1loss/FBt1/assignment_loss‚0.24563177387838095 1loss/FBt2/assignment_loss‚0.36664489617808727 .loss/FRt3/detection_loss‚1.747717914439864 /loss/FRt4/detection_loss‚3.4279788553714754 1loss/SRqqt3/detection_loss‚0.8818826821533222 trainer/global_step‚6499 /loss/FRt1/detection_loss‚2.5784037147249492 2loss/SRqqt3/assignment_loss‚1.1293632195950531 1loss/SRqqt4/detection_loss‚2.6607098443941637 0loss/FRt1/assignment_loss‚2.9545499716486248 _step‚534 0loss/FBt3/detection_loss‚0.39800314969033934 .loss/FRt2/assignment_loss‚4.10380295301095 2loss/SRqqt1/assignment_loss‚1.0046089954483204 2loss/SRqqt2/assignment_loss‚1.5999011313628957 2loss/SRqqt4/assignment_loss‚1.3528293418613346 /loss/FBt2/detection_loss‚2.4264267661741803 %loss/total_loss‚42.64374473822524 _runtime‚7563.947420981 .loss/FRt2/detection_loss‚2.305450045515616߇ó?b©Ü 2loss/SRqqt2/assignment_loss‚1.5999011313628957 2loss/SRqqt4/assignment_loss‚1.3528293418613346 /loss/FBt2/detection_loss‚2.4264267661741803 %loss/total_loss‚42.64374473822524 _runtime‚7563.947420981 .loss/FRt2/detection_loss‚2.305450045515616 0loss/FRt3/assignment_loss‚1.2748859777288921 1loss/FBt4/assignment_loss‚0.37616956494743475 0loss/SRqqt1/detection_loss‚1.823259257353269 0loss/SRqqt2/detection_loss‚2.331010630837193 /loss/FBt1/detection_loss‚1.8175299485524496 1loss/FBt3/assignment_loss‚0.07785017136400067 epoch‚5 % _timestamp‚1.7884715107627172e+09 /loss/FBt4/detection_loss‚2.3795459449291227 /loss/FRt4/assignment_loss‚3.379587987065315 1loss/FBt1/assignment_loss‚0.24563177387838095 1loss/FBt2/assignment_loss‚0.36664489617808727 .loss/FRt3/detection_loss‚1.747717914439864 /loss/FRt4/detection_loss‚3.4279788553714754 1loss/SRqqt3/detection_loss‚0.8818826821533222 trainer/global_step‚6499 /loss/FRt1/detection_loss‚2.5784037147249492 2loss/SRqqt3/assignment_loss‚1.1293632195950531 1loss/SRqqt4/detection_loss‚2.6607098443941637 0loss/FRt1/assignment_loss‚2.9545499716486248 _step‚534 0loss/FBt3/detection_loss‚0.39800314969033934 .loss/FRt2/assignment_loss‚4.10380295301095 2loss/SRqqt1/assignment_loss‚1.0046089954483204–ð¡p’ª‚(:‡ áÑçÔôôÀ‡ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚124.321' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.08025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:BPÎZ«‚(:Ï áÑçÔÖ†Þˆ network.sent‚ 1.6670596e+07 disk.sdb1.out‚0 disk.nvme0n1.out‚ 42320.0625 disk.sdb2.out‚828.75 disk.sda.in‚ 4.40234375 disk.nvme0n1.in‚ 329776.46875 disk.md0.in‚8.890625 disk.md0.out‚ 798.90234375) proc.memory.percent‚18.49515438079834 disk.sdb.in‚ 4.48828125' proc.memory.availableMB‚ 10298.65625 disk.sda.out‚828.75 disk.sda2.in‚ 4.40234375 disk.sda2.out‚828.75 disk.sdb2.in‚ 4.48828125% memory_percent‚37.141990661621094# proc.memory.rssMB‚ 3030.24609375 network.recv‚ 2.60638e+06) disk./.usagePercent‚83.16188790507111 disk.sdb1.in‚0 disk.dm-0.out‚97690.76171875 disk.rbd2.out‚ 1.962890625 disk.dm-0.in‚530976.41015625 disk.sda1.out‚0 disk.rbd0.out‚ 1.962890625 cpu‚25.18872771844394$ disk./.usageGB‚2974.271926879883 disk.rbd2.in‚0# disk.nvme1n1.in‚201199.94140625# disk.nvme1n1.out‚55370.69921875 disk.sdb.out‚828.75 disk.rbd0.in‚0.015625 proc.cpu.threads‚17 disk.sda1.in‚0 disk.md0p1.in‚8.890625 disk.md0p1.out‚ 798.90234375ÝB‰¬‚(:… ðÑçÔÚÂч gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚139.868' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.967 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²¼A™‚­‚(:÷ ðÑçÔàŒˆ proc.cpu.threads‚17) disk./.usagePercent‚83.16188694512631% disk./.usageGB‚2974.2718925476074! disk.sda2.out‚830.03857421875 disk.sdb2.in‚ 4.48828125) proc.memory.percent‚18.49515438079834 disk.dm-0.in‚530976.41015625 disk.sda1.in‚0 disk.md0.out‚ 800.1328125 disk.rbd0.in‚0.015625 disk.nvme0n1.in‚ 329776.46875# disk.nvme0n1.out‚42320.11328125 network.recv‚ 2.61334e+06 disk.sda2.in‚ 4.40234375 disk.rbd0.out‚ 1.962890625 disk.rbd2.in‚0# disk.nvme1n1.in‚201199.94140625 disk.sda.in‚ 4.40234375 network.sent‚ 1.6767716e+07 disk.md0p1.out‚ 800.1328125 disk.sdb.in‚ 4.48828125 disk.sdb1.out‚0 cpu‚25.18839357381942 disk.dm-0.out‚97794.25390625 disk.sda1.out‚0* proc.memory.availableMB‚10298.65234375# proc.memory.rssMB‚ 3030.24609375 disk.rbd2.out‚ 1.962890625! disk.nvme1n1.out‚ 55474.140625 disk.sda.out‚830.03857421875% memory_percent‚37.142014503479004! disk.sdb2.out‚830.03857421875 disk.sdb.out‚830.03857421875 disk.sdb1.in‚0 disk.md0p1.in‚8.890625 disk.md0.in‚8.890625–‡.h’®‚(:‡ ÿÑçÔáñý… gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚140.145' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.03625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#€¯‚(:õ ÿÑçÔû²„‡ disk.sda.out‚832.27490234375% disk./.usageGB‚2973.6287422180176 disk.sda1.out‚0! disk.sda2.out‚832.27490234375 disk.md0p1.in‚8.890625 disk.rbd2.in‚0 disk.rbd2.out‚ 1.962890625! disk.nvme0n1.out‚ 42320.328125 disk.sda.in‚ 4.40234375 network.sent‚ 1.6770746e+07) disk./.usagePercent‚83.14390419273165 disk.dm-0.in‚530976.41015625! disk.sdb2.out‚832.27490234375 disk.sdb.in‚ 4.48828125) proc.memory.availableMB‚ 10298.6484375 network.recv‚ 2.61711e+06 disk.sda2.in‚ 4.40234375 disk.md0.in‚8.890625 disk.sdb.out‚832.27490234375 disk.sdb1.in‚0) proc.memory.percent‚18.49515438079834% memory_percent‚37.142038345336914# proc.memory.rssMB‚ 3030.24609375# disk.nvme1n1.out‚55519.37890625 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 1.962890625# disk.nvme1n1.in‚201199.94140625 disk.nvme0n1.in‚ 329776.46875 cpu‚25.188096386067983 disk.dm-0.out‚97839.70703125 disk.md0p1.out‚ 802.296875 disk.md0.out‚ 802.296875 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sdb2.in‚ 4.48828125 proc.cpu.threads‚17ÃÑ¥´2°- _wandbruntime‚7614  _runtime‚7614º ª ı¾ % _timestamp‚1.7884715611416023e+09 _runtime‚ 7614.32651267 _step‚535 % lr-AdamW/pg1‚0.000424186096322569 % lr-AdamW/pg2‚0.000424186096322569 trainer/global_step‚6549âPeDɲà _step‚535 % lr-AdamW/pg1‚0.000424186096322569 % lr-AdamW/pg2‚0.000424186096322569 trainer/global_step‚6549 % _timestamp‚1.7884715611416023e+09 _runtime‚ 7614.32651267—·¦<þ2³- _wandbruntime‚7614  _runtime‚7614‚ Ô î'´èO [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13014536080384 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 178075233 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚282611055659520 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚681 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚282611079599616 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12977643718104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚279826499569664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47275572 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 % _timestamp‚1.7884715611425755e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚266811981492224 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74441648 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 178075233 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚253888719703120 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚279826499569664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚279826751241216 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 178101577 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 _runtime‚7614.327562285 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚295847259710976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124559069 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47276253 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚266811981492224 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124559069 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 121717903 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚683 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74441650 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚253888736742480 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚279826751241216 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13014536080384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚266811963489280 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13014769748992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 302660648 trainer/global_step‚6549 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 124559071 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13014769748992 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚266866609441668]Ðôû‰%µíO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚683 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74441650 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚253888736742480 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚279826751241216 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13014536080384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚266811963489280 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13014769748992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 302660648 trainer/global_step‚6549 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 124559071 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13014769748992 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚266866609441668 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13014536080384 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 178075233 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚282611055659520 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚681 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚282611079599616 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12977643718104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚279826499569664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47275572 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 % _timestamp‚1.7884715611425755e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚266811981492224 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74441648 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 178075233 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚253888719703120 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚279826499569664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚279826751241216 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 178101577 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 _runtime‚7614.327562285 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚295847259710976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124559069 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.ùÓi8jall.current‚ 251671552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47276253 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚266811981492224 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124559069 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 121717903˜û4÷32¶- _wandbruntime‚7615  _runtime‚7615tkßø&·òM A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 302679269 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13016500106240 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 178101925 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚282652763569664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚266851267529216 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 124577346 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 121735311 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47282859 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884715620884027e+09 _step‚537 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚266851249526272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74452450 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13016500106240 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚253926267099904 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178101925 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 124577344 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 178128295 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 302705641 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚654 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12979602207090 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 302679269 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74452452 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13016733788160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 302705641 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚279867749632512 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚295890941404672 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121734655 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124577344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚266851267529216 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 124577346 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 121735311 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47282859 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884715620884027e+09 _step‚537 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚266851249526272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74452450 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13016500106240 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚253926267099904 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178101925 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 124577344 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 178128295 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 302705641 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚654 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12979602207090 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 302679269 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74452452 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13016733788160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 302705641 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚279867749632512 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚295890941404672 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121734655 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124577344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13335937762304 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚268776265338880 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13112751787008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13075810739696 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚658 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚268776265338880 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚281889017125888 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 179413185 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 304912350 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 125472821 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47622394 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚656 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚281889017125888 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚268840819332932 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 304886004 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚298031215864320 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚284695254161920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚268776283341824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚281889017125888 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚268840819332932 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 304886004 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚298031215864320 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚284695254161920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚268776283341824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13335937762304 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚268776265338880 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13112751787008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13075810739696 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚658 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚268776265338880 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚281889017125888 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 179413185 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 304912350 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 125472821 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47622394 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚656 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0œÄ—C2Ê- _wandbruntime‚7669  _runtime‚7669'¬cbø&ËòM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13077540247598 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚268815569378816 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13114715812864 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚281930267188736 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13337636247552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13114949494784 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13114715812864 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚298074895217152 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13077769228890 _runtime‚7669.898157543 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚281930518873600 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 179466247 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚541 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚281930518873600 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚668 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13337933145088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚666 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚284736962072064 % _timestamp‚1.7884716167106628e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 304957343 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122621793 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13114949494784 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 304930971 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚298074574379520 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47629243 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125491096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚268880308179342 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 304930971 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125491094 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 179466247 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 304957343 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608$9í@˜Ì÷M PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 304957343 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13077540247598 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚268815569378816 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13114715812864 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚281930267188736 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13337636247552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13114949494784 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13114715812864 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚298074895217152 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13077769228890 _runtime‚7669.898157543 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚281930518873600 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 179466247 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚541 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚281930518873600 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚668 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13337933145088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚666 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚284736962072064 % _timestamp‚1.7884716167106628e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADcZ‘eeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 304957343 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122621793 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13114949494784 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 304930971 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚298074574379520 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47629243 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125491096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚268880308179342 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 304930971 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125491094 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 179466247 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370?VrY2Í- _wandbruntime‚7669  _runtime‚7669bM 4XÎÒ /loss/FRt1/assignment_loss‚2.984788115431623 1loss/SRqqt1/assignment_loss‚1.397791305387562 .loss/FRt2/detection_loss‚2.178241708314336 .loss/FRt3/detection_loss‚1.774837159684726 1loss/SRqqt3/detection_loss‚0.9718379139980321 0loss/SRqqt4/detection_loss‚2.456007404760881 .loss/FBt2/detection_loss‚2.298802524805069 _step‚542 0loss/FRt3/assignment_loss‚1.4132135989410537 /loss/FRt2/assignment_loss‚2.813578222249005 0loss/FBt3/assignment_loss‚0.0124891789043003 .loss/FRt1/detection_loss‚2.618703033139066 .loss/FRt4/detection_loss‚3.039305119410805 1loss/SRqqt2/detection_loss‚2.4050364955779044 /loss/FBt3/detection_loss‚0.6174415765064103 % _timestamp‚1.7884716167146764e+09 1loss/FBt2/assignment_loss‚0.07338337261166217 epoch‚5 /loss/FRt4/assignment_loss‚4.028378134188444 3loss/FBt1/assignment_loss‚0.0031263106012829668 2loss/SRqqt4/assignment_loss‚1.7665726831465056 2loss/FBt4/assignment_loss‚0.017821907583134414 1loss/SRqqt1/detection_loss‚1.9166983738541603 trainer/global_step‚6599 _runtime‚ 7669.89965487 2loss/SRqqt2/assignment_loss‚1.9253524647605034 2loss/SRqqt3/assignment_loss‚0.8668240252381532 .loss/FBt1/detection_loss‚3.387453335523605 /loss/FBt4/detection_loss‚2.1617391308148703 $loss/total_loss‚43.1294230954331Èrú ]Ï× 1loss/SRqqt2/detection_loss‚2.4050364955779044 /loss/FBt3/detection_loss‚0.6174415765064103 % _timestamp‚1.7884716167146764e+09 1loss/FBt2/assignment_loss‚0.07338337261166217 epoch‚5 /loss/FRt4/assignment_loss‚4.028378134188444 3loss/FBt1/assignment_loss‚0.0031263106012829668 2loss/SRqqt4/assignment_loss‚1.7665726831465056 2loss/FBt4/assignment_loss‚0.017821907583134414 1loss/SRqqt1/detection_loss‚1.9166983738541603 trainer/global_step‚6599 _runtime‚ 7669.89965487 2loss/SRqqt2/assignment_loss‚1.9253524647605034 2loss/SRqqt3/assignment_loss‚0.8668240252381532 .loss/FBt1/detection_loss‚3.387453335523605 /loss/FBt4/detection_loss‚2.1617391308148703 $loss/total_loss‚43.1294230954331 /loss/FRt1/assignment_loss‚2.984788115431623 1loss/SRqqt1/assignment_loss‚1.397791305387562 .loss/FRt2/detection_loss‚2.178241708314336 .loss/FRt3/detection_loss‚1.774837159684726 1loss/SRqqt3/detection_loss‚0.9718379139980321 0loss/SRqqt4/detection_loss‚2.456007404760881 .loss/FBt2/detection_loss‚2.298802524805069 _step‚542 0loss/FRt3/assignment_loss‚1.4132135989410537 /loss/FRt2/assignment_loss‚2.813578222249005 0loss/FBt3/assignment_loss‚0.0124891789043003 .loss/FRt1/detection_loss‚2.618703033139066 .loss/FRt4/detection_loss‚3.039305119410805žX^ {‘Ђ(:† ÊÒçÔöÙŠ‡ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.966' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.9915 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0YF@mÑ‚(:â ÊÒçÔÞÞÇ" disk.nvme1n1.out‚ 55831.3359375 disk.sda1.out‚0 disk.md0p1.out‚810.0625* proc.memory.availableMB‚10298.94140625# disk.nvme0n1.out‚42627.37890625 disk.dm-0.out‚98458.71484375 disk.sda2.in‚ 4.40234375 disk.sdb.out‚ 840.3359375 disk.sdb1.in‚0 disk.dm-0.in‚530976.41015625 disk.md0.out‚810.0625 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 1.962890625) proc.memory.percent‚18.49515438079834 cpu‚25.18642614615787 disk.sdb1.out‚0 disk.sda.in‚ 4.40234375 disk.sda1.in‚0 disk.md0p1.in‚8.890625) disk./.usagePercent‚83.17046831823816# disk.nvme1n1.in‚201199.94140625 disk.rbd2.in‚0# proc.memory.rssMB‚ 3030.24609375 network.recv‚ 2.641474e+06 disk.sdb2.in‚ 4.48828125 disk.sdb2.out‚ 840.3359375 disk.md0.in‚8.890625 disk.rbd2.out‚ 1.962890625$ memory_percent‚37.14025020599365 disk.sdb.in‚ 4.48828125 disk.sda.out‚ 840.3359375 proc.cpu.threads‚17 disk.nvme0n1.in‚ 329776.46875 disk.sda2.out‚ 840.3359375 network.sent‚ 1.6902383e+07% disk./.usageGB‚2974.5788040161133HÑð°Ò‚(:… ÙÒçÔü…è… gpu.0.gpu‚14.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚143.872' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.968 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€÷HƒÓ‚(:ø ÙÒçÔŒ«Ý† network.recv‚ 2.648613e+06% disk./.usageGB‚2974.5788040161133 disk.sda.in‚ 4.40234375 disk.dm-0.out‚ 98569.703125 disk.md0p1.in‚8.890625* proc.memory.availableMB‚10298.55859375 network.sent‚ 1.7009958e+07) disk./.usagePercent‚83.17046831823816 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.sdb1.in‚0$ memory_percent‚37.14258670806885 disk.md0p1.out‚ 811.38671875! disk.sdb2.out‚841.71142578125 disk.md0.in‚8.890625 disk.rbd0.out‚ 1.962890625" disk.nvme1n1.out‚ 55898.0703125 disk.sdb1.out‚0 proc.cpu.threads‚17# disk.nvme1n1.in‚201199.94140625) proc.memory.percent‚18.49515438079834 disk.sda1.in‚0 disk.nvme0n1.in‚ 329776.46875" disk.nvme0n1.out‚ 42671.6328125 disk.dm-0.in‚530976.41015625 disk.md0.out‚ 811.38671875 cpu‚25.186134459448297 disk.sda1.out‚0 disk.sdb.out‚841.71142578125! disk.sda2.out‚841.71142578125 disk.sda.out‚841.71142578125 disk.sda2.in‚ 4.40234375 disk.sdb2.in‚ 4.48828125 disk.rbd2.out‚ 1.962890625 disk.sdb.in‚ 4.48828125# proc.memory.rssMB‚ 3030.24609375nL€.’Ô‚(:‡ èÒçÔú§¤† gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚137.083' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.27075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0# Á‚Õ‚(:÷ èÒçÔ™•‡ disk.rbd0.in‚0.015625 disk.sdb1.in‚0 disk.dm-0.out‚ 98813.7578125 disk.sda1.in‚0) proc.memory.percent‚18.49515438079834 cpu‚25.185803630716823 disk.sdb2.in‚ 4.48828125! disk.sdb2.out‚843.65771484375 disk.sdb1.out‚0 disk.sda1.out‚0 disk.md0p1.in‚8.890625 disk.md0p1.out‚ 813.27734375 proc.cpu.threads‚17 network.sent‚ 1.7016004e+07 network.recv‚ 2.652856e+06 disk.md0.out‚ 813.27734375" disk.nvme1n1.out‚ 56031.1484375 disk.sdb.out‚843.65771484375! disk.sda2.out‚843.65771484375( proc.memory.availableMB‚ 10298.546875 disk.md0.in‚8.890625 disk.rbd2.out‚ 1.962890625# disk.nvme1n1.in‚201199.94140625 disk.sdb.in‚ 4.48828125 disk.sda.out‚843.65771484375 disk.sda2.in‚ 4.40234375! disk.nvme0n1.out‚ 42782.609375 disk.dm-0.in‚530976.5703125) disk./.usagePercent‚83.17743058447242$ memory_percent‚37.14265823364258# proc.memory.rssMB‚ 3030.24609375$ disk./.usageGB‚2974.827808380127 disk.rbd0.out‚ 1.962890625 disk.rbd2.in‚0 disk.sda.in‚ 4.40234375# disk.nvme0n1.in‚329776.62890625@Žÿ2Ö- _wandbruntime‚7719  _runtime‚7719A§NîÇ×Á & lr-AdamW/pg1‚0.0004122858606423358 & lr-AdamW/pg2‚0.0004122858606423358 trainer/global_step‚6649 % _timestamp‚1.7884716660972009e+09 _runtime‚7719.282151737 _step‚5431—zÌØÆ & lr-AdamW/pg1‚0.0004122858606423358 & lr-AdamW/pg2‚0.0004122858606423358 trainer/global_step‚6649 % _timestamp‚1.7884716660972009e+09 _runtime‚7719.282151737 _step‚543Ÿ¥"r¡2Ù- _wandbruntime‚7719  _runtime‚7719S(¸Dî'ÚèO M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126386569 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚283951786353664 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126386569 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚300215238420992 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13435462808576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚672 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 322948096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 307137706 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚283951786353664 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 180777481 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13173748780204 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚283951534682112 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚270740585191424 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚674 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13211201162240 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75521850 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 299008000 trainer/global_step‚6649 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 123489789 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚286779476604416 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚286779452664320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚257641755445520 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126386571 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126386571 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47968613 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 180777481 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 307164052 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75521848 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 % _timestamp‚1.7884716660983047e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚270740567188480 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13210967493632 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚270815275244640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚283951534682112 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13173519799120 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13210967493632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚300214915472896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 180751137 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249f˜ :$ÛíO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 180751137 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚257641772484880 _runtime‚7719.283455573 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 _step‚544 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13211201162240 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 180777481 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 307164052 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75521848 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 % _timestamp‚1.7884716660983047e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚270740567188480 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13210967493632 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚270815275244640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚283951534682112 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13173519799120 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13210967493632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚300214915472896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 180751137 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126386569 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚283951786353664 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126386569 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚300215238420992 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13435462808576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚672 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 322948096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 307137706 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚283951786353664 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 180777481 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13173748780204 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚283951534682112 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚270740585191424 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚674 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13211201162240 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75521850 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 299008000 trainer/global_step‚6649 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 123489789 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚286779476604416 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚286779452664320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚257641755445520 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126386571 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126386571 XKDeviceStatsMonitor.on_train_batch_start/inactiv ¯ôs¹e_split.small_pool.allocated‚47968613 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚6649 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47974583 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚270779871228416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚283992784744960 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚270779871228416 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13212931519488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123507233 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚270854764091050 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚257679302842304 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13213165201408 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126404846 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 307209045 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 180804199 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 180777829 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126404844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚283993036429824 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75532652 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13175707269398 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 180804199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚690 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚283993036429824 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126404846 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13212931519488 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75532650 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13437748597760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚6649 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47974583 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚270779871228416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚283992784744960 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚270779871228416 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13212931519488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123507233 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚270854764091050 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚257679302842304 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13213165201408 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126404846 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 307209045 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 180804199 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 180777829 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126404844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚283993036429824 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75532652 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13175707269398 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 180804199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚690 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚283993036429824 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126404846 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13212931519488 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75532650 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13437748597760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚286014052238336 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚286014303909888 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚272704887041024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 182115433 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76061950 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 278036480 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_ÞÞćpool.freed‚13271457839628 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚272704869038080 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13309416868864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48313382 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 127300319 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚272704887041024 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚259518273316720 % _timestamp‚1.7884717208220968e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13309183200256 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13309416868864 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13309183200256 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚666 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚288863675106816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚272789977176792 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚302398999319040 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚6699 _step‚548 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚286014052238336 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 182115433 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 127300321 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚288863651166720 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 309415754 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 124375330 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚302399301295616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 309389408 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 182089089 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13535626188800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48314046 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13535348152320 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 127300319 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚272704869038080 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 124375996 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚272789731156348 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301976576 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13535626188800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48314046 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13535348152320 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 127300319 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚272704869038080 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 124375996 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚272789731156348 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301976576 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚286014052238336 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚286014303909888 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚272704887041024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 182115433 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76061950 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 278036480 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13271457839628 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚272704869038080 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13309416868864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48313382 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 127300319 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚272704887041024 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚259518273316720 % _timestamp‚1.7884717208220968e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13309183200256 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13309416868864 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13309183200256 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚666 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚288863675106816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚272789977176792 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚302398999319040 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚6699 _step‚548 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚286014052238336 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 182115433 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 127300321 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚288863651166720 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 309415754 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 124375330 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚302399301295616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104¤CD 2ð- _wandbruntime‚7774  _runtime‚7774: M¶ö&ñðM ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13311380908032 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13273416328614 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76072752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚288905335136768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13311147226112 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 182115781 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13537631025152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚6699 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13311147226112 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48320311 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚272829220002758 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 127318594 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 309460747 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚272829466023410 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚259555803674144 # _timestamp‚1.78847172177068e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚272744173078016 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 _step‚549 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13311380908032 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13537348807680 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚302442683944448 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚272744173078016 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚286055302301184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 309434375 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124393061 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76072750 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 182142151 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚286055553986048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48320984 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚259555820713504 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 127318596 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675-Ö`ð;òõM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 _step‚549 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13311380908032 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13537348807680 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚302442683944448 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚272744173078016 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚286055302301184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 309434375 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124393061 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76072750 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 182142151 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚286055553986048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48320984 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚259555820713504 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 SEDeviceSŒ ÑÌÀtatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 127318596 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13311380908032 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13273416328614 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76072752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚288905335136768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13311147226112 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 182115781 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13537631025152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚6699 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13311147226112 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48320311 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚272829220002758 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 127318594 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 309460747 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚272829466023410 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚259555803674144 # _timestamp‚1.78847172177068e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚272744173078016 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273¥7qÝG2ó- _wandbruntime‚7774  _runtime‚7774&ñá¯ZôÔ 0loss/FRt1/assignment_loss‚2.9385844755172728 epoch‚5 trainer/global_step‚6699 1loss/SRqqt4/assignment_loss‚1.619728121906519 2loss/FBt3/assignment_loss‚0.027415927752239364 _runtime‚7774.957496352 0loss/FRt3/assignment_loss‚1.0521825593378809 .loss/FBt1/detection_loss‚2.667299669981003 /loss/FRt1/detection_loss‚2.9310438013076783 _step‚550 /loss/FRt2/assignment_loss‚3.558806493094093 2loss/SRqqt3/assignment_loss‚0.8919837770717485 1loss/FBt1/assignment_loss‚0.19701582423806485 .loss/FRt4/detection_loss‚3.160343885421753 1loss/SRqqt1/detection_loss‚1.9693698980087457 1loss/SRqqt3/detection_loss‚0.7055075836323557 0loss/SRqqt4/detection_loss‚2.347800523042679 /loss/FBt3/detection_loss‚0.5641405343538127 1loss/FBt2/assignment_loss‚0.08970077971488727 0loss/FBt4/assignment_loss‚0.3270752609435779 .loss/FRt3/detection_loss‚1.299971049754984 $ _timestamp‚1.788471721772796e+09 2loss/SRqqt2/assignment_loss‚1.4873901563577163 /loss/FRt2/detection_loss‚2.3677157145116086 /loss/FBt2/detection_loss‚1.8812717356179889 /loss/FBt4/detection_loss‚1.7126248081525166 /loss/FRt4/assignment_loss‚3.362894127002129 2loss/SRqqt1/assignment_loss‚1.6438018915265105 /loss/SRqqt2/detection_loss‚2.18737465601701 &loss/total_loss‚40.991043254264774Ãý½ % lr-AdamW/pg2‚0.000400385575357866 trainer/global_step‚6749 # _timestamp‚1.78847177456737e+09 _runtime‚7827.752404308 _step‚551 % lr-AdamW/pg1‚0.000400385575357866^ªu Èþ % lr-AdamW/pg1‚0.000400385575357866 % lr-AdamW/pg2‚0.000400385575357866 trainer/global_step‚6749 # _timestamp‚1.78847177456737e+09 _runtime‚7827.752404308 _step‚551§Ëšû§2ÿ- _wandbruntime‚7827  _runtime‚7827Œ+5 í'€çO N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 125261547 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚288076821466112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚652 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 _step‚552 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 183427041 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76602050 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚650 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13369624861220 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76602048 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 311667456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48660149 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128214071 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚288076821466112 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚290947849669120 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13407632575488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚290947873609216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚288076569794560 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128214071 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13635065340928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 311641110 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚274669170887680 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 trainer/global_step‚6749 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48659499 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚274669188890624 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚261394808227280 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚304582915010048 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13635358057472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚304583231666688 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128214069 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13407632575488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚261394808227280 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚304582915010048 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13635358057472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚304583231666688 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128214069 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13407632575488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚288076569794560 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128214071 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13635065340928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 311641110 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚274669170887680 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 trainer/global_step‚6749 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.smaln¿*Íl_pool.freed‚48659499 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚274669188890624 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125278915 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚274708456924672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76612852 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 311686077 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128232344 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13409596614656 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128232346 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76612850 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48666065 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚288118071542272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚654 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚304626908168704 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 311686077 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚274708474927616 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚652 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128232344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 311712449 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13371583350414 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13409362932736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 _runtime‚7828.707683693™n›ý&„÷M I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚274708474927616 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚274708456924672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76612852 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 311686077 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128232344 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13409596614656 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128232346 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76612850 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48666065 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚288118071542272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚654 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚304626908168704 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 311686077 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚274708474927616 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚652 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128232344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 311712449 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13371583350414 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13409362932736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 _runtime‚7828.707683693 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 183480103 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125278915 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚276633490740224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚263271326098480 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129127821 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13505614613504 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚276738642979764 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚290139087350784 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 _runtime‚7879.595267389 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚293032072111616 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.larÀ 6ã ge_pool.freed‚293032048171520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49009732 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚306767136332288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚276738889000208 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13505848282112 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 184791337 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚276633472737280 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 129127821 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77142148 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 trainer/global_step‚6799 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚290139339022336 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚263271309059120 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚276633490740224 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13505848282112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚706 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 129127819 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13467562901728 _step‚556 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49009028 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚276633490740224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚263271326098480 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129127821 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13505614613504 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚276738642979764 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚290139087350784 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 _runtime‚7879.595267389 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚293032072111616 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚293032048171520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49009732 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚306767136332288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚276738889000208 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13505848282112 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 184791337 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚276633472737280 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 129127821 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77142148 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 trainer/global_step‚6799 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚290139339022336 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚263271309059120 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚276633490740224 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13505848282112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚706 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 129127819 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13467562901728 _step‚556 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49009028 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 trainer/global_step‚6799 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 313964151 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49016239 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 129146094 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13507812321280 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 129146094 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13469292409630 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77152952 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚276672758774272 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚290180589098496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129146096 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 184818055 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 184818055 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13507578639360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚306810810409472 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13736763721728 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚276672758774272 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13507812321280 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 126169887 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77152950 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚263308856455904 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚276672776777216 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 _runtime‚7880.544398061 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49016935 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚263308839416544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚698 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚276778131826174 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126169189 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚696 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 184791685 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13737054327808si§Ý˜÷M RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚276672776777216 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 _runtime‚7880.544398061 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49016935 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚263308839416544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚698 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚276778131826174 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126169189 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚696 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 184791685 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13737054327808 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 trainer/global_step‚6799 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 313964151 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49016239 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 129146094 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.à\œ¸ allocated‚13507812321280 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 129146094 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13469292409630 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77152952 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚276672758774272 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚290180589098496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129146096 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 184818055 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 184818055 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13507578639360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚306810810409472 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13736763721728 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚276672758774272 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13507812321280 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 126169887 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77152950 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚263308856455904­ÂŽã®2™- _wandbruntime‚7880  _runtime‚7880µv[šÕ /loss/FRt4/assignment_loss‚2.668770370276078 2loss/SRqqt3/assignment_loss‚1.0302493427052266 2loss/FBt2/assignment_loss‚0.001989233001950197 .loss/FRt1/detection_loss‚3.098345183663898 /loss/FRt3/detection_loss‚1.4603647530108455 0loss/SRqqt4/detection_loss‚2.476771792461132 /loss/FBt3/detection_loss‚0.4291706900112331 2loss/SRqqt2/assignment_loss‚1.6292971914732142 2loss/SRqqt4/assignment_loss‚1.5765955956845448 /loss/FBt4/detection_loss‚2.4562965035438538 &loss/total_loss‚44.264603766146976 $ _timestamp‚1.788471827360669e+09 trainer/global_step‚6799 /loss/FRt1/assignment_loss‚2.572499348057641 0loss/FRt3/assignment_loss‚1.3922107721131944 2loss/SRqqt1/assignment_loss‚1.3276038099201801 /loss/FBt1/detection_loss‚2.3728582014640174 0loss/SRqqt3/detection_loss‚0.957765617656211 0loss/FBt1/assignment_loss‚0.1353049096252145 1loss/SRqqt1/detection_loss‚1.8234867025976595 _runtime‚7880.545354486 /loss/FRt2/detection_loss‚2.2353309074357197 1loss/SRqqt2/detection_loss‚2.2017223019348946 .loss/FBt2/detection_loss‚5.194669057925542 /loss/FRt4/detection_loss‚3.1254787121130074 _step‚558 0loss/FBt3/assignment_loss‚0.1437777859855501 1loss/FBt4/assignment_loss‚0.25833552209658706 epoch‚5 /loss/FRt2/assignment_loss‚3.695709461389586ùBð`›Ú 0loss/FBt1/assignment_loss‚0.1353049096252145 1loss/SRqqt1/detection_loss‚1.8234867025976595 _runtime‚7880.545354486 /loss/FRt2/detection_loss‚2.2353309074357197 1loss/SRqqt2/detection_loss‚2.2017223019348946 .loss/FBt2/detection_loss‚5.194669057925542 /loss/FRt4/detection_loss‚3.1254787121130074 _step‚558 0loss/FBt3/assignment_loss‚0.1437777859855501 1loss/FBt4/assignment_loss‚0.25833552209658706 epoch‚5 /loss/FRt2/assignment_loss‚3.695709461389586 /loss/FRt4/assignment_loss‚2.668770370276078 2loss/SRqqt3/assignment_loss‚1.0302493427052266 2loss/FBt2/assignment_loss‚0.001989233001950197 .loss/FRt1/detection_loss‚3.098345183663898 /loss/FRt3/detection_loss‚1.4603647530108455 0loss/SRqqt4/detection_loss‚2.476771792461132 /loss/FBt3/detection_loss‚0.4291706900112331 2loss/SRqqt2/assignment_loss‚1.6292971914732142 2loss/SRqqt4/assignment_loss‚1.5765955956845448 /loss/FBt4/detection_loss‚2.4562965035438538 &loss/total_loss‚44.264603766146976 $ _timestamp‚1.788471827360669e+09 trainer/global_step‚6799 /loss/FRt1/assignment_loss‚2.572499348057641 0loss/FRt3/assignment_loss‚1.3922107721131944 2loss/SRqqt1/assignment_loss‚1.3276038099201801 /loss/FBt1/detection_loss‚2.3728582014640174 0loss/SRqqt3/detection_loss‚0.957765617656211®ÍŠf’œ‚(:‡ œÔçÔåõÕ† gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚141.087' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.27175 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù͈~‚(:ó œÔçÔ’Ž™‡$ disk./.usageGB‚2973.822929382324 disk.rbd2.in‚0# disk.nvme0n1.out‚43096.05859375 disk.sda2.in‚ 4.40234375 network.sent‚ 1.7391018e+07 disk.rbd0.in‚0.015625 disk.sdb1.in‚0# disk.nvme0n1.in‚329838.90234375 disk.sda2.out‚863.8544921875 disk.md0p1.out‚ 832.71875# proc.memory.rssMB‚ 3030.24609375 cpu‚25.181960648829175 disk.sdb.out‚863.8544921875 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.out‚ 2.041015625 disk.md0.out‚ 832.71875 disk.rbd0.out‚ 1.974609375" disk.nvme1n1.in‚201285.4296875# disk.nvme1n1.out‚58024.49609375 disk.dm-0.in‚531124.33203125 disk.md0p1.in‚8.890625 proc.cpu.threads‚17) disk./.usagePercent‚83.14933374711931 disk.sdb2.in‚ 4.48828125* proc.memory.availableMB‚10298.30078125 disk.sdb1.out‚0 disk.sda.in‚ 4.40234375$ memory_percent‚37.14416027069092) proc.memory.percent‚18.49515438079834 network.recv‚ 2.710224e+06 disk.sdb2.out‚863.8544921875 disk.md0.in‚8.890625 disk.sdb.in‚ 4.48828125 disk.sda.out‚863.8544921875 disk.dm-0.out‚101120.5546875€¾ÙRž‚(:‚ «ÔçÔÔÍû‡ gpu.0.gpu‚4.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚137.36' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.34 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢jI…Ÿ‚(:ú «ÔçÔ±‘êˆ network.recv‚ 2.717152e+06% disk./.usageGB‚2973.8559913635254" disk.nvme0n1.in‚329840.3359375$ memory_percent‚37.14413642883301) proc.memory.availableMB‚ 10298.3046875# proc.memory.rssMB‚ 3030.24609375) disk./.usagePercent‚83.15025817395122! disk.sda2.out‚865.22998046875 disk.sdb2.in‚ 4.48828125 disk.md0.out‚ 834.0390625 disk.sdb1.out‚0 cpu‚25.181605982434796 proc.cpu.threads‚17 disk.md0.in‚8.890625 disk.rbd0.out‚ 2.037109375 disk.sda.in‚ 4.40234375! disk.dm-0.out‚101234.92578125 disk.md0p1.out‚ 834.0390625 network.sent‚ 1.7498234e+07 disk.sda.out‚865.22998046875 disk.md0p1.in‚8.890625 disk.rbd2.out‚ 2.041015625! disk.nvme1n1.out‚ 58052.703125# disk.nvme0n1.out‚43182.22265625 disk.dm-0.in‚531128.94140625 disk.sdb.out‚865.22998046875 disk.sdb1.in‚0) proc.memory.percent‚18.49515438079834 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚ 4.40234375# disk.nvme1n1.in‚201288.60546875! disk.sdb2.out‚865.22998046875 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.sdb.in‚ 4.48828125˜ŠêW ‚(:… ºÔçÔ–•¼‡ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚87.451' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚21.86275 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôp¡~¡‚(:ó ºÔçÔ­ ûˆ disk.sda.out‚ 867.236328125 disk.dm-0.in‚531128.94140625 disk.sdb1.out‚0 disk.sda2.in‚ 4.40234375 disk.sdb2.in‚ 4.48828125 disk.sdb2.out‚ 867.236328125# disk.nvme1n1.in‚201288.60546875) proc.memory.percent‚18.49515438079834 proc.cpu.threads‚17" disk.nvme0n1.in‚329840.3359375 disk.rbd0.out‚ 2.037109375* proc.memory.availableMB‚10298.32421875) disk./.usagePercent‚83.15653834608254% disk./.usageGB‚2974.0806007385254 disk.dm-0.out‚ 101318.125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚ 867.236328125 disk.md0p1.out‚ 835.97265625# proc.memory.rssMB‚ 3030.24609375 network.sent‚ 1.750474e+07 disk.sda.in‚ 4.40234375# disk.nvme0n1.out‚43187.83984375 disk.rbd0.in‚0.015625 disk.sdb1.in‚0 disk.md0p1.in‚8.890625 disk.md0.out‚ 835.97265625$ memory_percent‚37.14401721954346 network.recv‚ 2.722014e+06 disk.sdb.in‚ 4.48828125 disk.md0.in‚8.890625# disk.nvme1n1.out‚58130.28515625 disk.rbd2.in‚0 disk.rbd2.out‚ 2.12841796875 disk.sdb.out‚ 867.236328125 cpu‚25.181262688928317‡…^Õ2¢- _wandbruntime‚7934  _runtime‚7934u¨@ɣà _step‚559 ' lr-AdamW/pg1‚0.00038849515995879777 ' lr-AdamW/pg2‚0.00038849515995879777 trainer/global_step‚6849 % _timestamp‚1.7884718814895897e+09 _runtime‚7934.674817437˜ö_nÎ¤È ' lr-AdamW/pg1‚0.00038849515995879777 ' lr-AdamW/pg2‚0.00038849515995879777 trainer/global_step‚6849 % _timestamp‚1.7884718814895897e+09 _runtime‚7934.674817437 _step‚559¯|D4Ï2¥- _wandbruntime‚7934  _runtime‚7934 U›mî'¦èO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 316144514 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚278597792589824 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13834860541440 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 186129289 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 127039151 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13604063988736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚292201604907008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130041569 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚672 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13565500942236 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77682250 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚308950818693120 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13565271961152 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 trainer/global_step‚6849 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 186102945 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49357573 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚278713098891472 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚278713344911916 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _runtime‚7934.675946592 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 130041571 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13604063988736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚295116246673920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚292201856578560 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 316170860 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 127039823 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚308951131155456 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130041571 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚292201856578560 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13603830320128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚278597774586880 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚278597792589824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13603830320128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚670 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7884718814908648e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 186102945 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 71345111040ž±qj!§íO LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13604063988736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚292201604907008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130041569 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚672 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13565500942236 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77682250 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚308950818693120 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13565271961152 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 trainer/global_step‚6849 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 186102945 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49357573 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚278713098891472 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚278713344911916 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _runtime‚7934.675946592 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 130041571 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13604063988736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚295116246673920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚292201856578560 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 316170860 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 127039823 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚308951131155456 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130041571 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚292201856578560 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13603830320128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚278597774586880 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚278597792589824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13603830320128 QCD)ëËN‰eviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚670 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7884718814908648e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 186102945 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 316144514 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚278597792589824 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13834860541440 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 186129289 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 127039151 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0°L©Å¨‚(:… ÉÔçÔ¢ªÌ† gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚141.704' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.426 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×ãØïx©‚(:í ÉÔçÔ̬í‡ disk.dm-0.in‚531128.94140625 disk.sda1.in‚0 network.sent‚ 1.7523409e+07 network.recv‚ 2.725978e+06 disk.md0p1.out‚ 837.515625 disk.sdb.out‚ 868.8359375 disk.sda.in‚ 4.40234375 disk.dm-0.out‚101647.3828125) disk./.usagePercent‚83.16351394488329 disk.md0p1.in‚8.890625 disk.sdb1.out‚0 cpu‚25.180897295613708 disk.sda.out‚ 868.8359375 disk.sda1.out‚0* proc.memory.availableMB‚10298.31640625 disk.rbd2.in‚0 disk.sdb2.in‚ 4.48828125 disk.md0.out‚ 837.515625$ memory_percent‚37.14406490325928 disk.sdb.in‚ 4.48828125 disk.sdb1.in‚0# disk.nvme0n1.out‚43387.57421875 disk.rbd2.out‚ 2.12841796875 disk.sdb2.out‚ 868.8359375 disk.md0.in‚8.890625# disk.nvme1n1.out‚58259.80859375" disk.nvme0n1.in‚329840.3359375 disk.sda2.out‚ 868.8359375# proc.memory.rssMB‚ 3030.24609375 proc.cpu.threads‚17 disk.rbd0.out‚ 2.1162109375# disk.nvme1n1.in‚201288.60546875 disk.sda2.in‚ 4.40234375) proc.memory.percent‚18.49515438079834% disk./.usageGB‚2974.3300819396973 disk.rbd0.in‚0.015625IãÙf2ª- _wandbruntime‚7935  _runtime‚7935µB#/ø&«òM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚292243106654720 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13606028027904 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77693050 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13567459431430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130059844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚265185357287744 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚7935.672781016 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13606028027904 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 186129637 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13605794345984 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 127057783 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚295157930643968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 316189481 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚635 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚633 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚265185374327104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49364731 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13836575997952 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚278752587737882 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚292242854969856 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130059844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚292243106654720 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _step‚561 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 186129637 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 316215853 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚308994506641920 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚278637078626816 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13605794345984 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 186156007 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 278023168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚308994808605184 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77693052 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚295157954584064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 316189481 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127057148 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130059846 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 % _timestamp‚1.7884718824870267e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚278637060623872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚278637060623872 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚6849 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301963264 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚278752833758534 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0#ê6ý&¬÷M J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚278752587737882 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚292242854969856 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130059844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚292243106654720 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _step‚561 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 186129637 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 316215853 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚308994506641920 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚278637078626816 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13605794345984 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 186156007 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 278023168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚308994808605184 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77693052 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚295157954584064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 316189481 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127057148 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130059846 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 % _timestamp‚1.7884718824870267e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚278637060623872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚278637060623872 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚6849 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301963264 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚278752833758534 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚292243106654720 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13606028027904 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77693050 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13567459431430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130059844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚265185357287744 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚7935.672781016 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13606028027904 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 186129637 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13605794345984 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 127057783 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚295157930643968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 316189481 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚635 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚633 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚265185374327104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49364731 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13836575997952±²"³|2­- _wandbruntime‚7935  _runtime‚7935m7mÁ^®Ø /loss/FRt1/assignment_loss‚2.516324818134308 2loss/SRqqt1/assignment_loss‚1.8231778115014288 0loss/SRqqt1/detection_loss‚2.015133782904199 0loss/FBt3/detection_loss‚0.45966840751640414 .loss/FBt4/detection_loss‚2.302089664671156 trainer/global_step‚6849 0loss/FRt4/assignment_loss‚3.4979643635451794 %loss/total_loss‚41.81183339371177 epoch‚5 1loss/FBt1/assignment_loss‚0.23065733054483478 2loss/FBt2/assignment_loss‚0.057839831935889006 .loss/FRt1/detection_loss‚2.805365330832345 2loss/SRqqt3/assignment_loss‚1.1659726905326049 2loss/SRqqt4/assignment_loss‚2.0839621767401697 .loss/FRt3/detection_loss‚1.268420402203565 1loss/SRqqt4/detection_loss‚2.5536873269081117 % _timestamp‚1.7884718824901962e+09 _step‚562 0loss/FRt2/assignment_loss‚3.4193528967025952 1loss/FBt4/assignment_loss‚0.30916404894861355 .loss/FRt2/detection_loss‚2.069471946762254 1loss/SRqqt2/detection_loss‚2.1000277490719506 1loss/SRqqt3/detection_loss‚1.1955265600482623 0loss/FRt3/assignment_loss‚1.2099348194357278 /loss/FBt1/detection_loss‚1.7543852220882068 1loss/FBt3/assignment_loss‚0.08802021510648911 .loss/FRt4/detection_loss‚2.748345802538097 _runtime‚7935.675101367 2loss/SRqqt2/assignment_loss‚1.4959700636201254 /loss/FBt2/detection_loss‚2.6413701314192553µ³ ‡c¯Ý 1loss/FBt3/assignment_loss‚0.08802021510648911 .loss/FRt4/detection_loss‚2.748345802538097 _runtime‚7935.675101367 2loss/SRqqt2/assignment_loss‚1.4959700636201254 /loss/FBt2/detection_loss‚2.6413701314192553 /loss/FRt1/assignment_loss‚2.516324818134308 2loss/SRqqt1/assignment_loss‚1.8231778115014288 0loss/SRqqt1/detection_loss‚2.015133782904199 0loss/FBt3/detection_loss‚0.45966840751640414 .loss/FBt4/detection_loss‚2.302089664671156 trainer/global_step‚6849 0loss/FRt4/assignment_loss‚3.4979643635451794 %loss/total_loss‚41.81183339371177 epoch‚5 1loss/FBt1/assignment_loss‚0.23065733054483478 2loss/FBt2/assignment_loss‚0.057839831935889006 .loss/FRt1/detection_loss‚2.805365330832345 2loss/SRqqt3/assignment_loss‚1.1659726905326049 2loss/SRqqt4/assignment_loss‚2.0839621767401697 .loss/FRt3/detection_loss‚1.268420402203565 1loss/SRqqt4/detection_loss‚2.5536873269081117 % _timestamp‚1.7884718824901962e+09 _step‚562 0loss/FRt2/assignment_loss‚3.4193528967025952 1loss/FBt4/assignment_loss‚0.30916404894861355 .loss/FRt2/detection_loss‚2.069471946762254 1loss/SRqqt2/detection_loss‚2.1000277490719506 1loss/SRqqt3/detection_loss‚1.1955265600482623 0loss/FRt3/assignment_loss‚1.2099348194357278 /loss/FBt1/detection_loss‚1.7543852220882068²‡ëgT’°‚(:‡ ØÔçÔÈãù† gpu.0.gpu‚39.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.787' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚27.44675 gpu.0.smClock‚1230 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÔpÐu|±‚(:ñ ØÔçÔ´€±ˆ disk.md0.out‚ 838.82421875 network.sent‚ 1.7619894e+07 disk.sda2.out‚ 870.19921875" disk.nvme0n1.in‚329840.3359375 disk.sdb2.out‚ 870.19921875 disk.rbd2.out‚ 2.19140625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0 disk.md0p1.in‚8.890625# disk.nvme1n1.out‚58311.63671875 disk.sda.out‚ 870.19921875 disk.dm-0.in‚531128.94140625$ memory_percent‚37.14430332183838# proc.memory.rssMB‚ 3030.24609375 disk.sdb.in‚ 4.48828125! disk.dm-0.out‚101699.26953125$ disk./.usageGB‚2974.831066131592 disk.rbd2.in‚0 disk.sda.in‚ 4.40234375 disk.md0p1.out‚ 838.82421875 disk.rbd0.in‚0.015625* proc.memory.availableMB‚10298.27734375 disk.sdb2.in‚ 4.48828125) disk./.usagePercent‚83.17752167256684 disk.sda1.out‚0 disk.sda2.in‚ 4.40234375 disk.rbd0.out‚ 2.1162109375 disk.sdb.out‚ 870.19921875" disk.nvme0n1.out‚ 43387.6328125) proc.memory.percent‚18.49515438079834 network.recv‚ 2.732818e+06 disk.md0.in‚8.890625# disk.nvme1n1.in‚201288.60546875 cpu‚25.180553941856637 proc.cpu.threads‚17º•Ä䜲‚(:‘ çÔçÔÃí ‡ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚119.978' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.994499999999995 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 W:‡³‚(:ü çÔçÔåÌô‡! disk.sdb2.out‚871.68798828125 disk.md0.in‚8.890625# disk.nvme1n1.in‚201288.60546875# disk.nvme1n1.out‚58389.44140625 disk.sda.in‚ 4.40234375" disk.nvme0n1.out‚ 43521.0234375 disk.sda2.in‚ 4.40234375 disk.sdb.in‚ 4.48828125 disk.sdb.out‚871.68798828125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.out‚0) disk./.usagePercent‚83.19152929358987 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 2.17919921875 disk.sda.out‚871.68798828125! disk.dm-0.out‚101910.46484375 disk.md0p1.out‚ 840.2578125* proc.memory.availableMB‚10298.61328125$ disk./.usageGB‚2975.332046508789 disk.rbd2.in‚0 disk.dm-0.in‚531128.94140625 disk.sda1.in‚0 cpu‚25.18024983211226! disk.sda2.out‚871.68798828125% memory_percent‚37.142252922058105 network.sent‚ 1.7703544e+07 disk.md0.out‚ 840.2578125 disk.rbd2.out‚ 2.19140625 disk.md0p1.in‚8.890625) proc.memory.percent‚18.49515438079834 proc.cpu.threads‚17 network.recv‚ 2.739082e+06 disk.sdb2.in‚ 4.48828125" disk.nvme0n1.in‚329840.3359375# proc.memory.rssMB‚ 3030.246093750 ,œ´‚(:‘ öÔçÔêÌ«‡ gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.455' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚27.363749999999996 gpu.0.smClock‚1245 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0֮ᅵ‚(:ú öÔçÔÐŒçˆ) disk./.usagePercent‚83.19147287016835 disk.sdb2.out‚873.5693359375 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 2.17919921875# disk.nvme1n1.out‚58518.62890625 disk.sda1.out‚0 network.sent‚ 1.7712144e+07 disk.sdb.out‚873.5693359375 disk.sda.in‚ 4.40234375 disk.sda2.in‚ 4.40234375 disk.sdb2.in‚ 4.48828125 disk.rbd2.out‚ 2.25439453125 disk.sdb1.in‚0 disk.sda2.out‚873.5693359375 disk.md0p1.out‚ 842.078125 proc.cpu.threads‚17 disk.md0.out‚ 842.078125$ memory_percent‚37.14444637298584* proc.memory.availableMB‚10298.25390625# proc.memory.rssMB‚ 3030.24609375) proc.memory.percent‚18.49515438079834 disk.md0.in‚8.890625" disk.nvme0n1.in‚329840.3359375! disk.dm-0.out‚102239.12109375 disk.md0p1.in‚8.890625 cpu‚25.179906707742553 disk.sdb.in‚ 4.48828125 disk.sda1.in‚0 network.recv‚ 2.743747e+06% disk./.usageGB‚2975.3300285339355 disk.rbd2.in‚0# disk.nvme1n1.in‚201288.60546875 disk.sdb1.out‚0 disk.sda.out‚873.5693359375" disk.nvme0n1.out‚ 43720.4921875 disk.dm-0.in‚531128.94140625´ Ko2¶- _wandbruntime‚7990  _runtime‚7990‚5¡É·Ã _runtime‚7990.363882894 _step‚563 ' lr-AdamW/pg1‚0.00038255675128319623 ' lr-AdamW/pg2‚0.00038255675128319623 trainer/global_step‚6899 % _timestamp‚1.7884719371789608e+09lü<Î¸È _runtime‚7990.363882894 _step‚563 ' lr-AdamW/pg1‚0.00038255675128319623 ' lr-AdamW/pg2‚0.00038255675128319623 trainer/global_step‚6899 % _timestamp‚1.7884719371789608e+09³QŒÆÔ2¹- _wandbruntime‚7990  _runtime‚7990jõÍA|ºèO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚311135092494848 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚311134775838208 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 % _timestamp‚1.7884719371796412e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13702046026752 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13934330661888 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚297200469116416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13663438982744 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 318396216 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚294264122463232 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚280687800823624 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13663210001660 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚280562094439424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚267024344801520 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13702279695360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13702279695360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚294264374134784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 318422562 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚280687554803180 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚267024361840880 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚294264122463232 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚280687554803180 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚267024361840880 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚294264122463232 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚311135092494848 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚311134775838208 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 % _timestamp‚1.7884719371796412e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13702046026752 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13934330661888 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚297200469116416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13663438982744 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 318396216 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚294264122463232 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚280687800823624 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13663210001660 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920´ 2¼- _wandbruntime‚7991  _runtime‚7991ŽÆvAø&½òM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 318467555 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13704010052608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78233150 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 127945854 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 318441183 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 318467555 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78233152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49712702 _runtime‚7991.450810431 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127945194 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚311178451614208 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 187493959 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚658 % _timestamp‚1.7884719382640514e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13704010052608 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚280601362473472 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49712044 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚311178770354688 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚267061892198304 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 187493959 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13704243734528 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚297242129146368 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _step‚565 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚280601362473472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130973596 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13936322467840 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130973596 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 318441183 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚294305624210944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 187467589 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚280727043649590 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0ì(+N¾÷M KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 318467555 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13704010052608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78233150 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 127945854 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 318441183 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 318467555 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78233152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49712702 _runtime‚7991.450810431 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127945194 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚311178451614208 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 187493959 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚658 % _timestamp‚1.7884719382640514e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13704010052608 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚280601362473472 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49712044 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚311178770354688 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚267061892198304 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 187493959 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13704243734528 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚297242129146368 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _step‚565 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚280601362473472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130973596 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13936322467840 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130973596 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 318441183 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚294305624210944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 187467589 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚280727043649590µii”S2¿- _wandbruntime‚7991  _runtime‚7991p&- YÀÓ _runtime‚7991.452173078 /loss/FRt2/detection_loss‚1.9730357711758608 0loss/FRt4/assignment_loss‚2.8701568091357195 1loss/SRqqt1/assignment_loss‚1.677268329907108 trainer/global_step‚6899 .loss/FRt3/detection_loss‚1.276123754924438 -loss/FBt4/detection_loss‚2.90096578001976 2loss/SRqqt3/assignment_loss‚1.0585412738942785 1loss/FBt4/assignment_loss‚0.13305052561917563 1loss/SRqqt2/detection_loss‚2.7512787437957265 1loss/SRqqt4/detection_loss‚2.5311970423007835 /loss/FBt1/detection_loss‚2.2304760267337165 0loss/FRt3/assignment_loss‚1.1651862185509478 .loss/FRt4/detection_loss‚3.081490597239247 /loss/FBt3/detection_loss‚0.4514782711318353 _step‚566 2loss/SRqqt2/assignment_loss‚1.1779233981733737 0loss/FBt1/assignment_loss‚0.2575058490357757 1loss/SRqqt3/detection_loss‚0.6844329512825138 epoch‚5 % _timestamp‚1.7884719382672198e+09 0loss/FRt1/assignment_loss‚2.6474793615012335 %loss/total_loss‚41.45832347856286 2loss/SRqqt4/assignment_loss‚1.4320529881094037 1loss/FBt3/assignment_loss‚0.07177674277335876 /loss/FBt2/detection_loss‚2.2910815314813093 /loss/FRt2/assignment_loss‚3.147316726354452 0loss/FBt2/assignment_loss‚0.2035756828013911 .loss/FRt1/detection_loss‚3.110224954013167 0loss/SRqqt1/detection_loss‚2.334704148608285³@î^ÁØ /loss/FBt2/detection_loss‚2.2910815314813093 /loss/FRt2/assignment_loss‚3.147316726354452 0loss/FBt2/assignment_loss‚0.2035756828013911 .loss/FRt1/detection_loss‚3.110224954013167 0loss/SRqqt1/detection_loss‚2.334704148608285 _runtime‚7991.452173078 /loss/FRt2/detection_loss‚1.9730357711758608 0loss/FRt4/assignment_loss‚2.8701568091357195 1loss/SRqqt1/assignment_loss‚1.677268329907108 trainer/global_step‚6899 .loss/FRt3/detection_loss‚1.276123754924438 -loss/FBt4/detection_loss‚2.90096578001976 2loss/SRqqt3/assignment_loss‚1.0585412738942785 1loss/FBt4/assignment_loss‚0.13305052561917563 1loss/SRqqt2/detection_loss‚2.7512787437957265 1loss/SRqqt4/detection_loss‚2.5311970423007835 /loss/FBt1/detection_loss‚2.2304760267337165 0loss/FRt3/assignment_loss‚1.1651862185509478 .loss/FRt4/detection_loss‚3.081490597239247 /loss/FBt3/detection_loss‚0.4514782711318353 _step‚566 2loss/SRqqt2/assignment_loss‚1.1779233981733737 0loss/FBt1/assignment_loss‚0.2575058490357757 1loss/SRqqt3/detection_loss‚0.6844329512825138 epoch‚5 % _timestamp‚1.7884719382672198e+09 0loss/FRt1/assignment_loss‚2.6474793615012335 %loss/total_loss‚41.45832347856286 2loss/SRqqt4/assignment_loss‚1.4320529881094037 1loss/FBt3/assignment_loss‚0.07177674277335876¶$óq ’‚(:‡ …ÕçÔÈÜч gpu.0.gpu‚28.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚142.313' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.57825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 «ê¿zÂ(:ï …ÕçÔ¨¥‰ disk.sdb1.in‚0 disk.sda.in‚ 4.40234375 disk.sdb2.in‚ 4.48828125# disk.nvme1n1.in‚201288.60546875 network.recv‚ 2.747851e+06) disk./.usagePercent‚83.17672182523647 disk.sdb.out‚ 875.310546875 disk.sda2.out‚ 875.310546875 proc.cpu.threads‚17" disk.nvme0n1.in‚329840.3359375 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 2.25439453125 disk.sdb1.out‚0 disk.sdb2.out‚ 875.310546875 disk.rbd0.out‚ 2.2421875$ memory_percent‚37.14408874511719& proc.memory.availableMB‚ 10298.3125 network.sent‚ 1.7728631e+07 disk.sda.out‚ 875.310546875 disk.nvme0n1.out‚ 43920.03125 disk.md0p1.in‚8.890625$ disk./.usageGB‚2974.802459716797 disk.sdb.in‚ 4.48828125! disk.dm-0.out‚102482.29296875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚ 4.40234375 disk.md0p1.out‚ 843.73046875 disk.md0.out‚ 843.73046875 disk.rbd2.in‚0# disk.nvme1n1.out‚58562.26171875) proc.memory.percent‚18.49515438079834 cpu‚25.179598203431933 disk.dm-0.in‚531128.94140625 disk.md0.in‚8.890625# proc.memory.rssMB‚ 3030.24609375Ă(:‘ ”ÕçÔØýÖ‡ gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.455' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚27.363749999999996 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02, €Å‚(:õ ”ÕçÔ•½‰ disk.dm-0.out‚102682.5078125 disk.md0.out‚ 845.015625 network.recv‚ 2.754736e+06$ disk./.usageGB‚2974.792995452881! disk.sda2.out‚876.64306640625 disk.md0p1.in‚8.890625 disk.md0p1.out‚ 845.015625) disk./.usagePercent‚83.17645720045627 disk.dm-0.in‚531128.94140625 disk.sda1.out‚0 disk.sdb2.in‚ 4.48828125 disk.rbd0.out‚ 2.2421875 disk.md0.in‚8.890625 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.sdb.out‚876.64306640625* proc.memory.availableMB‚10298.57421875# proc.memory.rssMB‚ 3030.24609375 proc.cpu.threads‚17 disk.sda2.in‚ 4.40234375 disk.sdb1.in‚0$ memory_percent‚37.14249134063721 disk.sdb.in‚ 4.48828125 network.sent‚ 1.7825794e+07 disk.sda.out‚876.64306640625 disk.sda1.in‚0! disk.sdb2.out‚876.64306640625 disk.rbd2.out‚ 2.25439453125! disk.nvme1n1.out‚ 58562.984375) proc.memory.percent‚18.49515438079834# disk.nvme1n1.in‚201288.60546875 disk.sdb1.out‚0 cpu‚25.17926446231334 disk.sda.in‚ 4.40234375" disk.nvme0n1.in‚329840.3359375" disk.nvme0n1.out‚ 44119.5234375G|"Æ‚(:… £ÕçÔ°»¨‡ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚120.31' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.0775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+Üj„Ç‚(:ù £ÕçÔÃÕ܇! disk.sda2.out‚878.59130859375! disk.sdb2.out‚878.59130859375 disk.sdb.out‚878.59130859375 disk.sdb1.out‚0 disk.md0p1.out‚ 846.890625" disk.nvme0n1.in‚329840.3359375 disk.md0.in‚8.890625 disk.sda.out‚878.59130859375 cpu‚25.178939849724454 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 2.2421875# disk.nvme1n1.out‚58563.14453125 disk.sdb1.in‚0# proc.memory.rssMB‚ 3030.24609375# disk.nvme1n1.in‚201288.60546875 disk.sda.in‚ 4.40234375) proc.memory.percent‚18.49515438079834 network.sent‚ 1.7828848e+07$ disk./.usageGB‚2974.830020904541 disk.sda2.in‚ 4.40234375 disk.rbd2.out‚ 2.25439453125 disk.sdb.in‚ 4.48828125 network.recv‚ 2.758642e+06" disk.nvme0n1.out‚ 44393.1328125! disk.dm-0.out‚102956.27734375 disk.md0p1.in‚8.890625 disk.sdb2.in‚ 4.48828125* proc.memory.availableMB‚10298.44921875) disk./.usagePercent‚83.17749244758103 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.dm-0.in‚531128.94140625 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0.out‚ 846.890625$ memory_percent‚37.14325428009033ù|¾}‘È‚(:† ²ÕçÔ͵ä… gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚129.614' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.4035 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÍÅhpÉ‚(:å ²ÕçÔ͚⇠network.sent‚ 1.7837427e+07 proc.cpu.threads‚17! disk.nvme0n1.out‚ 44467.015625 disk.sdb2.out‚ 880.109375" disk.nvme1n1.out‚ 58563.2265625% memory_percent‚36.681437492370605* proc.memory.availableMB‚10374.11328125 disk.sda1.in‚0 disk.sda2.in‚ 4.40234375 disk.sda2.out‚ 880.109375 disk.sdb1.in‚0 disk.sdb1.out‚0 cpu‚25.17857616053217 network.recv‚ 2.76342e+06 disk.sda.in‚ 4.40234375 disk.sda.out‚ 880.109375 disk.rbd2.in‚0 disk.rbd2.out‚ 2.25439453125# disk.nvme1n1.in‚201288.60546875 disk.dm-0.in‚531128.94140625 disk.dm-0.out‚103030.2421875 disk.rbd0.out‚ 2.2421875 disk.sdb.out‚ 880.109375* proc.memory.percent‚18.015360832214355 disk.md0p1.in‚8.890625 disk.sdb2.in‚ 4.48828125 disk.md0.in‚8.890625# proc.memory.rssMB‚ 2951.63671875) disk./.usagePercent‚83.18098579332909% disk./.usageGB‚2974.9549598693848" disk.nvme0n1.in‚329840.3359375 disk.sda1.out‚0 disk.md0p1.out‚ 848.3671875 disk.md0.out‚ 848.3671875 disk.rbd0.in‚0.015625 disk.sdb.in‚ 4.48828125»˜d2Ê- _wandbruntime‚8046  _runtime‚8046'ò›ŠÉËà _runtime‚8046.717271636 _step‚567 ' lr-AdamW/pg1‚0.00037662452570772073 ' lr-AdamW/pg2‚0.00037662452570772073 trainer/global_step‚6949 % _timestamp‚1.7884719935322523e+09FaM4ÎÌÈ ' lr-AdamW/pg2‚0.00037662452570772073 trainer/global_step‚6949 % _timestamp‚1.7884719935322523e+09 _runtime‚8046.717271636 _step‚567 ' lr-AdamW/pg1‚0.00037662452570772073·@Y2Í- _wandbruntime‚8046  _runtime‚8046ÿ˜ê®î'ÎèO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 188778849 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚282526396289024 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 trainer/global_step‚6949 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 128814352 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚313318727517696 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚8046.718475771 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50051234 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚296326640019456 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚313319042077184 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚282526396289024 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚296326891691008 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚299284643678720 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13800261733376 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 131869071 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13761377023252 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚282526378286080 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 131869069 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13800261733376 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 128813682 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚668 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14034374458368 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚268900862672720 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 188805193 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13800495401984 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78762450 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚268900879712080 % _timestamp‚1.7884719935333564e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0_à¸ÇÄÏíO SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 188778849 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚282526396289024 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 trainer/global_step‚6949 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 128814352 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚313318727517696 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚8046.718475771 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50051234 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚296326640019456 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚313319042077184 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚282526396289024 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚296326891691008 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 674105945‘ûó/7 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚299284643678720 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13800261733376 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 131869071 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13761377023252 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚282526378286080 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 131869069 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13800261733376 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 128813682 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚668 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14034374458368 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚268900862672720 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 188805193 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13800495401984 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78762450 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚268900879712080 % _timestamp‚1.7884719935333564e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944¸ozò2Ð- _wandbruntime‚8047  _runtime‚8047¨ƒkˆø&ÑòM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 188831911 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 320719257 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚282565682326016 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 128832581 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚726 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚282701745581950 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚569 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13802459441152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚299326327648768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚282701499561298 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚296367890082304 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚296368141767168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13802459441152 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚268938410069504 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13763335512446 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13802225759232 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚724 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚299326351588864 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚6949 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 131887344 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 128831855 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14036367443456 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14036070545920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13802225759232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚313362398194688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚268938393030144 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚282565682326016 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 131887346 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131887344 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚282565664323072 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 320719257 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 320692885 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 188805541 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 320692885 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78773250 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884719945020618e+09è1|ßý&Ò÷M S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚296367890082304 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚296368141767168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13802459441152 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚268938410069504 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13763335512446 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13802225759232 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚724 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚299326351588864 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚6949 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 131887344 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 128831855 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14036367443456 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14036070545920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13802225759232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚313362398194688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚268938393030144 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚282565682326016 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 131887346 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131887344 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚282565664323072 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 320719257 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 320692885 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 188805541 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 320692885 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78773250 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884719945020618e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 188831911 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 320719257 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚282565682326016 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 128832581 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚726 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚282701745581950 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚569 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13802459441152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚299326327648768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚282701499561298 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0¹ ¸2Ó- _wandbruntime‚8047  _runtime‚8047†i3(ZÔÔ 1loss/FBt1/assignment_loss‚0.03549839366141643 1loss/SRqqt2/detection_loss‚2.5755332200615495 /loss/FRt4/detection_loss‚2.7518735621124506 2loss/SRqqt1/assignment_loss‚1.5311930278937023 1loss/SRqqt2/assignment_loss‚1.316989357272784 2loss/SRqqt3/assignment_loss‚0.9159938103454024 1loss/SRqqt4/assignment_loss‚1.520007560029626 1loss/SRqqt1/detection_loss‚2.3630172659953437 0loss/FRt1/assignment_loss‚3.1275980906053023 0loss/FRt4/assignment_loss‚3.1678329091519117 2loss/FBt3/assignment_loss‚0.005087906078164719 /loss/SRqqt4/detection_loss‚2.22667038962245 /loss/FBt4/detection_loss‚2.0752259913612816 _step‚570 1loss/SRqqt3/detection_loss‚0.9296869091983688 0loss/FBt3/detection_loss‚0.42909116432776734 1loss/FBt4/assignment_loss‚0.24304212572035974 .loss/FRt2/detection_loss‚2.213689546989785 .loss/FRt3/assignment_loss‚1.06454436007668 1loss/FBt2/assignment_loss‚0.12511317745732597 .loss/FRt1/detection_loss‚2.591421376575123 .loss/FBt1/detection_loss‚2.635939488808314 .loss/FBt2/detection_loss‚3.541787435611089 epoch‚5 trainer/global_step‚6949 $ _timestamp‚1.788471994503917e+09 /loss/FRt2/assignment_loss‚2.634529314257882 /loss/FRt3/detection_loss‚1.3282569881528616 &loss/total_loss‚41.349623371366945 _runtime‚8047.688605239ÍÌU_ÕÙ 0loss/FRt1/assignment_loss‚3.1275980906053023 0loss/FRt4/assignment_loss‚3.1678329091519117 2loss/FBt3/assignment_loss‚0.005087906078164719 /loss/SRqqt4/detection_loss‚2.22667038962245 /loss/FBt4/detection_loss‚2.0752259913612816 _step‚570 1loss/SRqqt3/detection_loss‚0.9296869091983688 0loss/FBt3/detection_loss‚0.42909116432776734 1loss/FBt4/assignment_loss‚0.24304212572035974 .loss/FRt2/detection_loss‚2.213689546989785 .loss/FRt3/assignment_loss‚1.06454436007668 1loss/FBt2/assignment_loss‚0.12511317745732597 .loss/FRt1/detection_loss‚2.591421376575123 .loss/FBt1/detection_loss‚2.635939488808314 .loss/FBt2/detection_loss‚3.541787435611089 epoch‚5 trainer/global_step‚6949 $ _timestamp‚1.788471994503917e+09 /loss/FRt2/assignment_loss‚2.634529314257882 /loss/FRt3/detection_loss‚1.3282569881528616 &loss/total_loss‚41.349623371366945 _runtime‚8047.688605239 1loss/FBt1/assignment_loss‚0.03549839366141643 1loss/SRqqt2/detection_loss‚2.5755332200615495 /loss/FRt4/detection_loss‚2.7518735621124506 2loss/SRqqt1/assignment_loss‚1.5311930278937023 1loss/SRqqt2/assignment_loss‚1.316989357272784 2loss/SRqqt3/assignment_loss‚0.9159938103454024 1loss/SRqqt4/assignment_loss‚1.520007560029626 1loss/SRqqt1/detection_loss‚2.3630172659953437º7wZ,Ö‚(:… ÁÕçÔÆŠ€‰ gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚128.388' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.097 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ2¢vׂ(:ë ÁÕçԆͼŠ disk.rbd2.out‚ 2.25439453125% disk./.usageGB‚2975.0799140930176 disk.dm-0.in‚531128.94140625 disk.sda.in‚ 4.40234375 disk.md0p1.in‚8.890625* proc.memory.percent‚18.015360832214355" disk.nvme0n1.in‚329840.3359375 disk.sdb2.out‚ 881.6640625# disk.nvme1n1.in‚201288.60546875# proc.memory.rssMB‚ 2951.63671875 cpu‚25.17825830945507 proc.cpu.threads‚17 network.sent‚ 1.7841129e+07 disk.sdb1.in‚0 disk.sda.out‚ 881.6640625 disk.rbd0.in‚0.015625) disk./.usagePercent‚83.18447956571927 disk.sdb.in‚ 4.48828125 disk.sdb.out‚ 881.6640625 disk.sda1.out‚0 disk.md0p1.out‚ 849.859375% memory_percent‚36.680054664611816! disk.dm-0.out‚103307.67578125 disk.sda2.in‚ 4.40234375 disk.md0.in‚8.890625* proc.memory.availableMB‚10374.33984375 disk.sda2.out‚ 881.6640625 disk.sdb2.in‚ 4.48828125 disk.rbd0.out‚ 2.2421875 disk.rbd2.in‚0 network.recv‚ 2.767991e+06 disk.sdb1.out‚0" disk.nvme0n1.out‚ 44744.1328125 disk.sda1.in‚0 disk.md0.out‚ 849.859375# disk.nvme1n1.out‚58563.54296875N^¥‘Ø‚(:† ÐÕçÔû¦¾‡ gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚138.302' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.5755 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬€§ïtÙ‚(:é ÐÕçÔ¼ï‡ disk.rbd0.in‚0.015625$ memory_percent‚36.68172359466553# proc.memory.rssMB‚ 2951.63671875 network.sent‚ 1.794791e+07 network.recv‚ 2.774147e+06 disk.dm-0.in‚531128.94140625 disk.sda2.out‚ 883.1171875 disk.nvme1n1.out‚ 58563.65625* proc.memory.percent‚18.015360832214355 proc.cpu.threads‚17 disk.md0.in‚8.890625 disk.sda.out‚ 883.1171875* proc.memory.availableMB‚10374.06640625 cpu‚25.17794522600754) disk./.usagePercent‚83.19149814871447% disk./.usageGB‚2975.3309326171875" disk.nvme0n1.out‚ 45024.6640625 disk.sdb2.out‚ 883.1171875# disk.nvme1n1.in‚201288.60546875 disk.sdb.in‚ 4.48828125 disk.sda.in‚ 4.40234375 disk.rbd2.out‚ 2.25439453125 disk.sdb.out‚ 883.1171875 disk.sda2.in‚ 4.40234375 disk.md0.out‚ 851.24609375 disk.dm-0.out‚103588.3203125 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0p1.in‚8.890625 disk.sdb2.in‚ 4.48828125 disk.rbd0.out‚ 2.2421875 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.sdb1.out‚0" disk.nvme0n1.in‚329840.3359375 disk.md0p1.out‚ 851.24609375æw·›Ú‚(: ßÕçÔŒÁŽ† gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚117.81' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.452499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Xyö…Û‚(:ú ßÕçÔÃõ©‡ disk.md0.out‚ 853.0546875# memory_percent‚36.6840124130249# proc.memory.rssMB‚ 2951.63671875* proc.memory.percent‚18.015360832214355 disk.rbd2.out‚ 2.25439453125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda.out‚884.97216796875 disk.rbd2.in‚0 network.recv‚ 2.778013e+06# disk.nvme0n1.out‚45301.99609375 disk.md0p1.out‚ 853.0546875 network.sent‚ 1.7955484e+07 disk.rbd0.out‚ 2.2421875) disk./.usagePercent‚83.16186262652498 disk.sda1.in‚0 disk.md0p1.in‚8.890625 disk.sdb2.in‚ 4.48828125! disk.sdb2.out‚884.97216796875 proc.cpu.threads‚17! disk.nvme1n1.out‚ 58563.765625 disk.sda.in‚ 4.40234375! disk.dm-0.out‚103865.76171875 disk.sda2.in‚ 4.40234375! disk.sda2.out‚884.97216796875 disk.md0.in‚8.890625 disk.rbd0.in‚0.015625 disk.sdb.in‚ 4.48828125$ disk./.usageGB‚2974.271022796631" disk.nvme0n1.in‚329840.3359375 disk.dm-0.in‚531128.94140625 disk.sdb.out‚884.97216796875* proc.memory.availableMB‚10373.69140625 cpu‚25.177650726928327 disk.sda1.out‚0# disk.nvme1n1.in‚201288.60546875ö+¼’Ü‚(:‡ îÕçÔ®Á‡ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚119.701' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.92525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïü ¦Ý‚(:ö îÕçÔ–­ö‡ disk.sda.out‚886.42333984375" disk.nvme0n1.out‚ 45568.8828125 disk.sda1.out‚0 disk.sdb1.out‚0! disk.dm-0.out‚104132.73046875 disk.sdb1.in‚0 proc.cpu.threads‚17! disk.sda2.out‚886.42333984375 disk.sda1.in‚0 disk.sdb.in‚ 4.48828125 disk.md0.in‚8.890625$ memory_percent‚36.68100833892822 cpu‚25.177349962447696 network.recv‚ 2.782471e+06 disk.sda.in‚ 4.40234375" disk.nvme0n1.in‚329840.3359375 disk.md0p1.out‚854.4375 disk.md0.out‚854.4375( disk./.usagePercent‚83.1618476940505 disk.sdb2.in‚ 4.48828125! disk.sdb2.out‚886.42333984375 disk.sdb.out‚886.42333984375* proc.memory.availableMB‚10374.18359375# proc.memory.rssMB‚ 2951.63671875* proc.memory.percent‚18.015360832214355 disk.sda2.in‚ 4.40234375% disk./.usageGB‚2974.2704887390137 network.sent‚ 1.7961839e+07 disk.dm-0.in‚531128.94140625 disk.md0p1.in‚8.890625 disk.rbd0.out‚ 2.2421875 disk.rbd2.in‚0 disk.rbd2.out‚ 2.25439453125 disk.rbd0.in‚0.015625# disk.nvme1n1.in‚201288.60546875# disk.nvme1n1.out‚58563.84765625Ìc—Þ- _wandb•2÷L!runtime‚8100  _runtime‚8100FâèLÇßÁ & lr-AdamW/pg1‚0.0003706997194992899 & lr-AdamW/pg2‚0.0003706997194992899 trainer/global_step‚6999 % _timestamp‚1.7884720468989098e+09 _runtime‚8100.083860309 _step‚571Û¢ ÌàÆ trainer/global_step‚6999 % _timestamp‚1.7884720468989098e+09 _runtime‚8100.083860309 _step‚571 & lr-AdamW/pg1‚0.0003706997194992899 & lr-AdamW/pg2‚0.0003706997194992899»ÂË+2á- _wandbruntime‚8100  _runtime‚8100zÑ÷î'âèO % _timestamp‚1.7884720469001126e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 132782819 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13898477440000 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 129701527 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚673 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚298389409247232 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 132782821 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 190143145 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 190116801 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14134126196736 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13898477440000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132782819 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 190143145 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _runtime‚8100.085154545 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 132782821 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚298389409247232 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 322925966 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 129700854 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13859315063760 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚270777397583280 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79302550 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273842176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚270777380543920 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚298389157575680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50398306 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚315502694535680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚298389157575680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚79302548 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50398977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚671 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚315502992317952 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 190143145 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 190116801 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14134126196736 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13898477440000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132782819 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 190143145 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _runtime‚8100.085154545 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 132782821 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚298389409247232 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 322925966 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 129700854 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13859315063760 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚270777397583280 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79302550 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273842176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚270777380543920 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚298389157575680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50398306 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚315502694535680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚298389157575680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚79302548 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50398977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚671 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚315502992317952 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 % _timestamp‚1.7884720478707962e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 190143493 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13900675147776 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚315546368944640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _step‚573 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132801096 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13900675147776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13861044571662 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129718080 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50404730 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚298430659323392 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 322970959 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚270814927940704 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 190169863 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚298430407638528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132801096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14135842793472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚284529966172672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79313352 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 129718768 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 190143493 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚686 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚301410550091264 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚688îjÑæöM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚284529984175616 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 322944587 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚284676201493658 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 trainer/global_step‚6999 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 322944587 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚298430659323392 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13900441465856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonit]þ¯ü+ or.on_train_batch_end/inactive_split_bytes.all.freed‚315546368944640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _step‚573 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132801096 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13900675147776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13861044571662 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129718080 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50404730 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚298430659323392 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 322970959 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚270814927940704 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 190169863 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚298430407638528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132801096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14135842793472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚284529966172672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79313352 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 129718768 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 190143493 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚686 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚301410550091264 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚688 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚270814910901344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13900441465856 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚284529984175616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _runtime‚ 8101.05617368 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚301410526151168 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13861273552954 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 % _timestamp‚1.7884720478707962e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 190143493 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13900675147776 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 P¿’î‚(:‡ ›Öçԫ箇 gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚132.399' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.09975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"ÇÑNï‚(:ö ›ÖçÔ—‹ˆ$ disk./.usageGB‚2974.238670349121 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 2.2421875 disk.rbd2.in‚0 disk.sdb.in‚ 4.48828125 disk.sda2.in‚ 4.40234375 disk.md0p1.out‚ 859.5390625 disk.sdb2.out‚891.7197265625 disk.sda1.out‚0 disk.md0p1.in‚8.890625$ memory_percent‚36.68313026428223# proc.memory.rssMB‚ 2951.63671875 proc.cpu.threads‚17 network.sent‚ 1.8080848e+07# disk.nvme1n1.in‚201288.60546875" disk.nvme0n1.in‚329840.3359375 disk.sdb2.in‚ 4.48828125 disk.md0.in‚8.890625) proc.memory.availableMB‚ 10373.8359375* proc.memory.percent‚18.015360832214355 disk.sdb.out‚891.7197265625 network.recv‚ 2.797893e+06 disk.rbd2.out‚ 2.25439453125 disk.sda.in‚ 4.40234375 disk.sda1.in‚0 disk.sda.out‚891.7197265625# disk.nvme0n1.out‚45940.56640625 disk.sda2.out‚891.7197265625 cpu‚25.176404164967263# disk.nvme1n1.out‚58564.69921875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.dm-0.out‚ 104505.265625) disk./.usagePercent‚83.16095803855207 disk.dm-0.in‚531128.94140625 disk.md0.out‚ 859.5390625@øå2ð- _wandbruntime‚8151  _runtime‚8151<çê’ÇñÁ & lr-AdamW/pg1‚0.0003647835673786371 & lr-AdamW/pg2‚0.0003647835673786371 trainer/global_step‚7049 % _timestamp‚1.7884720987935972e+09 _runtime‚8151.978692227 _step‚575sët ÌòÆ & lr-AdamW/pg2‚0.0003647835673786371 trainer/global_step‚7049 % _timestamp‚1.7884720987935972e+09 _runtime‚8151.978692227 _step‚575 & lr-AdamW/pg1‚0.0003647835673786371¿h7a¯2ó- _wandbruntime‚8151  _runtime‚8151y©Õî'ôèO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13996693146624 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 325177668 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚272653915454480 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚79842648 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 191454753 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133696569 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 299008000 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚300451926803456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚300451675131904 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚286454999988224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300451675131904 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50745242 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 130587180 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133696571 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14233662625280 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚286454981985280 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13957253104268 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚8151.980437805 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133696569 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13957024123184 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚303453064623616 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚710 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 191481097 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚286610922538304 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚7049 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 191481097 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚286454981985280 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚712 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚317686703308800 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14233961633280 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚286611168558748 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚300451926803456 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 191454753 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79842650 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 % _timestamp‚1.7884720987950919e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚286454999988224 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚303453040683520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 325151322 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 325177668 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚272653915454480 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚79842648 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 191454753 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133696569 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 299008000 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚300451926803456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚300451675131904 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚286454999988224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300451675131904 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50745242 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 130587180 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133696571 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14233662625280 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚286454981985280 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13957253104268 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚8151.980437805 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133696569 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13957024123184 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚303453064623616 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚710 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 191481097 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚286610922538304 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚7049 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 191481097 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚286454981985280 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚712 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚317686703308800 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14233961633280 YEDeviceStatsMonitor.on_train_batch_start/requested_bytesÕ—.all.allocated‚286611168558748 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚300451926803456 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 191454753 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79842650 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 % _timestamp‚1.7884720987950919e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚286454999988224 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚303453040683520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 325151322 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚286650411384714 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚286494268022272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 191507815 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13998657172480 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚300493176879616 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚300492925194752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79853450 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚317730695580672 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13998657172480 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 191481445 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚685 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚303494748593664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 trainer/global_step‚7049 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50751474 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 _step‚577 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13959211593462 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130604924 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚272691428772544 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13998890854400 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚8153.228263912 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚286494268022272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚317730389423104 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚286650657405366 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133714844 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚300493176879616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 325222661 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 325222661 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133714846 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚286494286025216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 325196289 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133714846 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13958982612170 $ _timestamp‚1.788472100041912e+09 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79853452 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130605611 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 191481445 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14235664769536 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚08èÎ6ü&øöM SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 _step‚577 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13959211593462 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130604924 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚272691428772544 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13998890854400 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚8153.228263912 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚286494268022272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚317730389423104 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚286650657405366 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133714844 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚300493176879616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 325222661 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 325222661 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133714846 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚286494286025216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 325196289 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133714846 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13958982612170 $ _timestamp‚1.788472100041912e+09 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79853452 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130605611 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 191481445 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14235664769536 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚286650411384714 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚286494268022272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 191507815 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13998657172480 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚300493176879616 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚300492925194752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79853450 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚317730695580672 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13998657172480 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 191481445 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚685 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚303494748593664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 trainer/global_step‚7049 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50751474 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273Á±‡ï2ù- _wandbruntime‚8153  _runtime‚8153Ÿ‚œµYúÓ 0loss/FRt3/assignment_loss‚1.4042713467417092 .loss/FRt2/detection_loss‚2.280214532734183 1loss/SRqqt3/detection_loss‚0.9819744360277948 /loss/FBt4/detection_loss‚2.6151622469936098 2loss/SRqqt3/assignment_loss‚1.0184823262226796 /loss/FRt1/detection_loss‚2.9121662840760987 %loss/total_loss‚41.41054776990582 0loss/FRt1/assignment_loss‚2.7341023662994646 0loss/FRt2/assignment_loss‚3.0148628871691856 1loss/FBt3/assignment_loss‚0.04472482232262839 *loss/FRt4/detection_loss‚ 2.90831742088 1loss/SRqqt2/assignment_loss‚1.573269536970435 1loss/FBt4/assignment_loss‚0.07263259521723901 1loss/SRqqt1/detection_loss‚2.4665674567222595 /loss/FBt1/detection_loss‚2.5449542850255966 /loss/FBt3/detection_loss‚0.4393402549291128 0loss/SRqqt2/detection_loss‚2.337860004321949 trainer/global_step‚7049 _runtime‚8153.229551148 2loss/SRqqt1/assignment_loss‚1.4604559057662565 2loss/SRqqt4/assignment_loss‚1.3368451216004111 1loss/SRqqt4/detection_loss‚2.5661609118635003 /loss/FBt2/detection_loss‚2.4003930898273693 % _timestamp‚1.7884721000448048e+09 /loss/FRt4/assignment_loss‚2.376505132516225 0loss/FBt1/assignment_loss‚0.2129180546874194 1loss/FBt2/assignment_loss‚0.14867297633266469 epoch‚5 _step‚578 /loss/FRt3/detection_loss‚1.5596937746580304‚Üo^ûØ 2loss/SRqqt3/assignment_loss‚1.0184823262226796 /loss/FRt1/detection_loss‚2.9121662840760987 %loss/total_loss‚41.41054776990582 0loss/FRt1/assignment_loss‚2.7341023662994646 0loss/FRt2/assignment_loss‚3.0148628871691856 1loss/FBt3/assignment_loss‚0.04472482232262839 *loss/FRt4/detection_loss‚ 2.90831742088 1loss/SRqqt2/assignment_loss‚1.573269536970435 1loss/FBt4/assignment_loss‚0.07263259521723901 1loss/SRqqt1/detection_loss‚2.4665674567222595 /loss/FBt1/detection_loss‚2.5449542850255966 /loss/FBt3/detection_loss‚0.4393402549291128 0loss/SRqqt2/detection_loss‚2.337860004321949 trainer/global_step‚7049 _runtime‚8153.229551148 2loss/SRqqt1/assignment_loss‚1.4604559057662565 2loss/SRqqt4/assignment_loss‚1.3368451216004111 1loss/SRqqt4/detection_loss‚2.5661609118635003 /loss/FBt2/detection_loss‚2.4003930898273693 % _timestamp‚1.7884721000448048e+09 /loss/FRt4/assignment_loss‚2.376505132516225 0loss/FBt1/assignment_loss‚0.2129180546874194 1loss/FBt2/assignment_loss‚0.14867297633266469 epoch‚5 _step‚578 /loss/FRt3/detection_loss‚1.5596937746580304 0loss/FRt3/assignment_loss‚1.4042713467417092 .loss/FRt2/detection_loss‚2.280214532734183 1loss/SRqqt3/detection_loss‚0.9819744360277948 /loss/FBt4/detection_loss‚2.6151622469936098ÂâЂ‘ü‚(:† ªÖçÔí gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚104.826' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚26.2065 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gÙŽzý‚(:ï ªÖçԃý‰ disk.md0.out‚ 861.33984375# disk.nvme1n1.in‚201288.60546875" disk.nvme0n1.in‚329840.3359375$ memory_percent‚36.68467998504639# proc.memory.rssMB‚ 2951.63671875 disk.sdb2.out‚ 893.58203125 disk.sda.out‚ 893.58203125% disk./.usageGB‚2974.3230743408203# disk.nvme1n1.out‚58564.81640625 disk.sda2.in‚ 4.40234375 network.recv‚ 2.802232e+06) disk./.usagePercent‚83.16331800948579 disk.rbd0.out‚ 2.2421875 disk.rbd2.in‚0 disk.sda.in‚ 4.40234375 disk.md0p1.in‚8.890625 disk.rbd0.in‚0.015625 disk.sdb.in‚ 4.48828125 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.availableMB‚10373.58203125 cpu‚25.17613277657624 network.sent‚ 1.8084923e+07 disk.md0p1.out‚ 861.33984375 disk.rbd2.out‚ 2.25439453125 disk.sdb.out‚ 893.58203125 disk.dm-0.in‚531128.94140625! disk.dm-0.out‚104694.75390625 disk.sdb2.in‚ 4.48828125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.nvme0n1.out‚ 46129.9375 proc.cpu.threads‚17 disk.md0.in‚8.890625 disk.sda2.out‚ 893.58203125* proc.memory.percent‚18.015360832214355f ¯ö‘þ‚(:† ¹ÖçԤߢ‡ gpu.0.gpu‚34.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚136.466' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.1165 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô‹v€ÿ‚(:õ ¹ÖçÔێˈ network.recv‚ 2.809405e+06 disk.sdb2.out‚ 895.193359375 disk.rbd0.out‚ 2.2421875# disk.nvme1n1.in‚201288.60546875 disk.sdb.out‚ 895.193359375" disk.nvme0n1.in‚329840.3359375! disk.dm-0.out‚104778.44140625 disk.md0.in‚8.890625 disk.md0.out‚ 862.8828125 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚36.68472766876221* proc.memory.percent‚18.015360832214355) disk./.usagePercent‚83.16347117400991 disk.sdb1.out‚0 disk.sda.out‚ 895.193359375 disk.md0p1.out‚ 862.8828125 cpu‚25.175847145122322 disk.sdb2.in‚ 4.48828125 disk.rbd2.in‚0 disk.sda.in‚ 4.40234375" disk.nvme0n1.out‚ 46213.3046875 disk.sda2.in‚ 4.40234375 network.sent‚ 1.8194708e+07 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 2.25439453125 disk.sdb.in‚ 4.48828125 disk.dm-0.in‚531128.94140625 disk.md0p1.in‚8.890625 proc.cpu.threads‚17% disk./.usageGB‚2974.3285522460938# disk.nvme1n1.out‚58565.13671875 disk.sdb1.in‚0* proc.memory.availableMB‚10373.57421875# proc.memory.rssMB‚ 2951.63671875 disk.sda2.out‚ 895.193359375cä߀ ‚(:… ÈÖçÔáû«‡ gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚142.036' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.509 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”Êç„ ‚(:ù ÈÖçÔíùˆ disk.md0.in‚8.890625 disk.rbd2.in‚0 disk.sda.out‚897.3603515625 disk.sda2.in‚ 4.40234375$ memory_percent‚36.68475151062012 network.recv‚ 2.813663e+06 disk.sdb2.out‚897.3603515625 disk.md0p1.in‚8.890625* proc.memory.percent‚18.015360832214355 disk.sda1.out‚0 disk.sdb2.in‚ 4.48828125 disk.md0.out‚ 864.97265625 network.sent‚ 1.8200769e+07 disk.rbd0.in‚0.015625 disk.dm-0.out‚ 104922.578125 disk.md0p1.out‚ 864.97265625# disk.nvme1n1.out‚58565.26953125# disk.nvme0n1.out‚46357.30859375 disk.dm-0.in‚531128.94140625 proc.cpu.threads‚17 disk.sda2.out‚897.3603515625# proc.memory.rssMB‚ 2951.63671875 disk.sda1.in‚0 disk.rbd0.out‚ 2.2421875 disk.rbd2.out‚ 2.25439453125# disk.nvme1n1.in‚201288.60546875 disk.sdb1.out‚0 disk.sda.in‚ 4.40234375" disk.nvme0n1.in‚329840.3359375 cpu‚25.175584777069847) disk./.usagePercent‚83.17047237133839% disk./.usageGB‚2974.5789489746094 disk.sdb.in‚ 4.48828125 disk.sdb.out‚897.3603515625 disk.sdb1.in‚0) proc.memory.availableMB‚ 10373.5703125XÚÍy‘‚ ‚(:† ×ÖçÔ÷ðð… gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚140.478' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.1195 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DË“L1ƒ ‚(:ø ×ÖçÔô††ˆ) disk./.usagePercent‚83.17747730178549 disk.sda1.out‚0 disk.rbd0.out‚ 2.2421875 disk.rbd2.out‚ 2.25439453125 disk.sdb.out‚898.7177734375 disk.sdb1.out‚0 disk.md0p1.in‚8.890625 disk.sda.out‚898.7177734375$ memory_percent‚36.68475151062012 network.recv‚ 2.817629e+06# proc.memory.rssMB‚ 2951.63671875 disk.sdb1.in‚0 disk.sda.in‚ 4.40234375# disk.nvme0n1.out‚46671.16796875 disk.dm-0.in‚531128.94140625 disk.sdb2.out‚898.7177734375 disk.sdb.in‚ 4.48828125) proc.memory.availableMB‚ 10373.5703125% disk./.usageGB‚2974.8294792175293 disk.sda1.in‚0 disk.md0p1.out‚ 866.28515625 disk.md0.out‚ 866.28515625# disk.nvme1n1.in‚201288.60546875* proc.memory.percent‚18.015360832214355 disk.sda2.in‚>flzR 4.40234375 disk.sda2.out‚898.7177734375 disk.sdb2.in‚ 4.48828125 disk.md0.in‚8.890625 disk.rbd2.in‚0# disk.nvme1n1.out‚58565.50390625 disk.dm-0.out‚ 105236.671875 network.sent‚ 1.8206044e+07 disk.rbd0.in‚0.015625" disk.nvme0n1.in‚329840.3359375 cpu‚25.17533158576122 proc.cpu.threads‚1782²2„ - _wandbruntime‚8208  _runtime‚8208â6 Æ… À trainer/global_step‚7099 $ _timestamp‚1.788472155513396e+09 _runtime‚8208.698349546 _step‚579 & lr-AdamW/pg1‚0.0003588773022629972 & lr-AdamW/pg2‚0.0003588773022629972¯¿B¸Ë† Å _runtime‚8208.698349546 _step‚579 & lr-AdamW/pg1‚0.0003588773022629972 & lr-AdamW/pg2‚0.0003588773022629972 trainer/global_step‚7099 $ _timestamp‚1.788472155513396e+09ÃL«ø2‡ - _wandbruntime‚8208  _runtime‚8208í¢¼í'ˆ çO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚302514192688128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚288585624470456 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚302514444359680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚319871034509312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 trainer/global_step‚7099 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚274530433325680 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚288419283834880 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14095142521856 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80382750 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚288585378450012 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 327429370 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚319870713658368 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚302514444359680 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14094908853248 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 192792705 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 131475879 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51093129 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚288419301837824 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14054962163692 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 134610319 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 192819049 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚80382748 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚302514192688128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14094908853248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14333771383296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 327403024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 192819049 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 _step‚580 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚288419301837824 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 % _timestamp‚1.7884721555144515e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 134610321 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 134610321 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 296910848 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚305537263126016 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14333474472448 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚274530416286320 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 % _timestamp‚1.7884721555144515e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 134610321 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 134610321 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 296910848 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚305537263126016 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14333474472448 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚274530416286320 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚288585624470456 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚302514444359680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚319871034509312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 trainer/global_step‚7099 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚274530433325680 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚288419283834880 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14095142521856 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80382750 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚288585378450012 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 327429370 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚319870713658368 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚302514444359680 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14094908853248 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 192792705 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 131475879 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51093129 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚288419301837824 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14054962163692 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 134610319 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 192819049 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚80382748 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚302514192688128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14094908853248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14333771383296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 327403024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 192819049 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0Ä÷‡”2Š - _wandbruntime‚8209  _runtime‚8209ÒÚÕÒø&‹ òM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚319914708492288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚288458587874816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51100603 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚302555442750976 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 _step‚581 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 192845767 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚288625113317074 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14097106561024 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚288624867296422 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 % _timestamp‚1.7884721566661658e+09 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚274567946643744 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14096872879104 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 134628594 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 327447991 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 134628596 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 327474363 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 327447991 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131493446 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚305578947096064 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚302555694435840 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80393552 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚274567963683104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _runtime‚8209.852684572 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 192819397 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚288458587874816 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14335466595840 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚288458569871872 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14096872879104 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14057149633970 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 192819397 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚707 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51099896 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14335761396224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚709 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚319914389751808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚288458569871872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134628594 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14097106561024 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134628596 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚288458587874816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_eΣ©“!nd/reserved_bytes.all.allocated‚ 7134511104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51100603 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚302555442750976 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 _step‚581 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 192845767 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚288625113317074 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14097106561024 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚288624867296422 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 % _timestamp‚1.7884721566661658e+09 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚274567946643744 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14096872879104 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 134628594 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 327447991 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 134628596 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 327474363 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 327447991 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131493446 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚305578947096064 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚302555694435840 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80393552 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚274567963683104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _runtime‚8209.852684572 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 192819397 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚288458587874816 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14335466595840 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚288458569871872 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14096872879104 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14057149633970 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 192819397 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚707 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51099896 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14335761396224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚709 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚319914389751808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚288458569871872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134628594 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14097106561024 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134628596 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚290383603687424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 % _timestamp‚1.7884722116319456e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14193124559872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚276406951196880 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚304576710244352 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 132361481 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 329681072 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51438633 _step‚584 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14153129185284 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14152900204200 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚290383585684480 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 194157001 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚290559834361720 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚290383603687424 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14193358228480 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚80922848 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14433515822080 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 194130657 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 194130657 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚290383603687424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 % _timestamp‚1.7884722116319456e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14193124559872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚276406951196880 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚304576710244352 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 132361481 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 329681072 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51438633 _step‚584 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14153129185284 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14152900204200 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚290383585684480 UGDeviceStats‡¿ØMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 194157001 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚290559834361720 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0Èvç’œ ‚(:‡ “×çÔŠ¹·‡ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚142.653' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.66325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷ÑòU} ‚(:ò “×çÔȶ ˆ cpu‚25.174273490473638 disk.sda1.in‚0 disk.rbd0.out‚ 2.2421875# disk.nvme1n1.out‚58566.24609375 disk.sdb.in‚ 4.48828125 disk.sdb2.in‚ 4.48828125$ memory_percent‚36.68532371520996 network.sent‚ 1.834223e+07 proc.cpu.threads‚17$ disk./.usageGB‚2974.756320953369 disk.sdb1.in‚0 disk.sda2.out‚905.3857421875 network.recv‚ 2.837234e+06 disk.rbd2.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 10373.4765625 disk.md0p1.out‚ 872.7109375 disk.md0.out‚ 872.7109375# disk.nvme1n1.in‚201288.60546875 disk.sda.out‚905.3857421875" disk.nvme0n1.in‚329840.3359375 disk.dm-0.in‚531128.94140625 disk.sdb2.out‚905.3857421875 disk.rbd2.out‚ 2.3408203125 disk.sdb1.out‚0 disk.sda.in‚ 4.40234375 disk.dm-0.out‚ 106142.3125 disk.sda2.in‚ 4.40234375 disk.sdb.out‚905.3857421875 disk.md0.in‚8.890625 disk.rbd0.in‚0.015625# disk.nvme0n1.out‚47576.06640625* proc.memory.percent‚18.015360832214355) disk./.usagePercent‚83.17543176610044 disk.md0p1.in‚8.890625# proc.memory.rssMB‚ 2951.63671875lN,&2ž - _wandbruntime‚8265  _runtime‚8265›åÛø&Ÿ òM OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚290599569228782 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 329726065 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 _runtime‚8265.791145894 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 % _timestamp‚1.7884722126055508e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚80933650 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚276444481554304 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 329699693 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚585 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14154858693186 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 194157349 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14435220469248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚290599323208130 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚304618211992064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 194157349 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚322098654576640 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚276444464514944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135542346 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 194183719 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚290422889724416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14195088585728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚290422871721472 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 trainer/global_step‚7149 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132379414 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚307663121658368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 194183719 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80933652 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51446432 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135542346 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 329726065 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14155087674478 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 329699693 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚322098342127616 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14195322267648 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14195088585728 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582èð%ý&  ÷M ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚322098342127616 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14195322267648 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14195088585728 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚290599569228782 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 329726065 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 _runtime‚8265.791145894 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 % _timestamp‚1.7884722126055508e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚80933650 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚276444481554304 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 329699693 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚585 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14154858693186 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 194157349 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14435220469248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚290599323208130 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚304618211992064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 194157349 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚322098654576640 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚276444464514944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135542346 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 194183719 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚290422889724416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14195088585728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚290422871721472 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 trainer/global_step‚7149 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132379414 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚307663121658368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 194183719 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80933652 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51446432 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135542346 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 329726065 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14155087674478 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 329699693ÉO€ï2¡ - _wandbruntime‚8265  _runtime‚8265WljŽ_¢ Ù 0loss/FRt1/assignment_loss‚2.8587611549430423 0loss/FRt2/assignment_loss‚3.0878325668541162 0loss/FRt4/assignment_loss‚3.0073519831611994 /loss/FRt3/detection_loss‚1.4145839208204831 1loss/SRqqt1/detection_loss‚1.8046050404393397 2loss/SRqqt4/assignment_loss‚1.7901830968481522 1loss/FBt3/assignment_loss‚0.18647381963322393 .loss/FBt4/detection_loss‚1.784857635912688 _step‚586 0loss/FBt1/assignment_loss‚0.2397208126150677 1loss/FBt4/assignment_loss‚0.27605077870309797 /loss/FRt4/detection_loss‚3.1478840226218816 /loss/FBt3/detection_loss‚0.6246641942397541 trainer/global_step‚7149 2loss/SRqqt1/assignment_loss‚1.3637448493716156 2loss/SRqqt3/assignment_loss‚1.0502709851338012 1loss/SRqqt2/detection_loss‚2.6093817937374113 1loss/SRqqt3/detection_loss‚0.8857070432921637 &loss/total_loss‚41.240887349110466 epoch‚5 % _timestamp‚1.7884722126074543e+09 .loss/FRt1/detection_loss‚2.670260819296042 1loss/SRqqt2/assignment_loss‚1.123076787739992 0loss/FBt2/assignment_loss‚0.3400428113069357 0loss/FRt3/assignment_loss‚1.2908720110143934 /loss/FBt2/detection_loss‚2.2468628862074445 _runtime‚8265.792132759 /loss/FRt2/detection_loss‚2.5860321881026156 /loss/FBt1/detection_loss‚2.5133606450898305 1loss/SRqqt4/detection_loss‚2.3383055020261696^âôd£ Þ 2loss/SRqqt3/assignment_loss‚1.0502709851338012 1loss/SRqqt2/detection_loss‚2.6093817937374113 1loss/SRqqt3/detection_loss‚0.8857070432921637 &loss/total_loss‚41.240887349110466 epoch‚5 % _timestamp‚1.7884722126074543e+09 .loss/FRt1/detection_loss‚2.670260819296042 1loss/SRqqt2/assignment_loss‚1.123076787739992 0loss/FBt2/assignment_loss‚0.3400428113069357 0loss/FRt3/assignment_loss‚1.2908720110143934 /loss/FBt2/detection_loss‚2.2468628862074445 _runtime‚8265.792132759 /loss/FRt2/detection_loss‚2.5860321881026156 /loss/FBt1/detection_loss‚2.5133606450898305 1loss/SRqqt4/detection_loss‚2.3383055020261696 0loss/FRt1/assignment_loss‚2.8587611549430423 0loss/FRt2/assignment_loss‚3.0878325668541162 0loss/FRt4/assignment_loss‚3.0073519831611994 /loss/FRt3/detection_loss‚1.4145839208204831 1loss/SRqqt1/detection_loss‚1.8046050404393397 2loss/SRqqt4/assignment_loss‚1.7901830968481522 1loss/FBt3/assignment_loss‚0.18647381963322393 .loss/FBt4/detection_loss‚1.784857635912688 _step‚586 0loss/FBt1/assignment_loss‚0.2397208126150677 1loss/FBt4/assignment_loss‚0.27605077870309797 /loss/FRt4/detection_loss‚3.1478840226218816 /loss/FBt3/detection_loss‚0.6246641942397541 trainer/global_step‚7149 2loss/SRqqt1/assignment_loss‚1.3637448493716156Ê­Þ¾’¤ ‚(:‡ ¢×çÔùÁЇ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚123.095' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.77375 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07Ò7‡¥ ‚(:ü ¢×çÔøü°‰ disk.dm-0.in‚531128.94140625 disk.sda1.out‚0# disk.nvme1n1.in‚201288.60546875 disk.sdb2.in‚ 4.48828125) disk./.usagePercent‚83.17745436977107# disk.nvme1n1.out‚58566.36328125 disk.sdb1.in‚0 disk.sdb1.out‚0" disk.nvme0n1.out‚ 47895.9921875 disk.md0p1.out‚ 874.51171875 disk.rbd0.in‚0.015625# memory_percent‚36.6840124130249$ disk./.usageGB‚2974.828659057617 disk.sda2.in‚ 4.40234375 proc.cpu.threads‚17 disk.sda.out‚907.24853515625" disk.nvme0n1.in‚329840.3359375! disk.sdb2.out‚907.24853515625 disk.md0.in‚8.890625 disk.md0.out‚ 874.51171875* proc.memory.percent‚18.015360832214355 network.sent‚ 1.8439026e+07 disk.rbd2.out‚ 2.40380859375 disk.md0p1.in‚8.890625 cpu‚25.17401199799202 network.recv‚ 2.844407e+06 disk.rbd2.in‚0 disk.sda.in‚ 4.40234375! disk.dm-0.out‚106462.35546875! disk.sda2.out‚907.24853515625 disk.rbd0.out‚ 2.2421875 disk.sdb.out‚907.24853515625 disk.sda1.in‚0* proc.memory.availableMB‚10373.69140625# proc.memory.rssMB‚ 2951.63671875 disk.sdb.in‚ 4.48828125£ à‘¦ ‚(:† ±×çÔ‰“õ† gpu.0.gpu‚6.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.357' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.83925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A1í§ ‚(:ô ±×çÔ—Ç disk.sdb.in‚ 4.48828125! disk.nvme0n1.out‚ 48060.171875 disk.md0p1.in‚8.890625! disk.sdb2.out‚908.61279296875 cpu‚25.17374045444965 disk.rbd0.in‚0.015625 network.recv‚ 2.851259e+06" disk.nvme0n1.in‚329840.3359375 disk.sda2.in‚ 4.40234375) disk./.usagePercent‚83.18094803550072 disk.dm-0.in‚531128.94140625 disk.sda1.in‚0 disk.rbd0.out‚ 2.2421875 disk.rbd2.out‚ 2.40380859375* proc.memory.availableMB‚10373.44140625* proc.memory.percent‚18.015360832214355 network.sent‚ 1.8525526e+07 disk.sdb1.out‚0 disk.sda.out‚908.61279296875 disk.sda1.out‚0 disk.md0p1.out‚ 875.796875 disk.md0.in‚8.890625 disk.md0.out‚ 875.796875$ memory_percent‚36.68553829193115% disk./.usageGB‚2974.9536094665527 disk.nvme1n1.out‚ 58566.53125 disk.sdb1.in‚0 disk.sda.in‚ 4.40234375 disk.sdb2.in‚ 4.48828125 disk.sdb.out‚908.61279296875 disk.dm-0.out‚ 106626.703125! disk.sda2.out‚908.61279296875# disk.nvme1n1.in‚201288.60546875 disk.rbd2.in‚0# proc.memory.rssMB‚ 2951.63671875 proc.cpu.threads‚17zX•Ä‘¨ ‚(:† À×çÔøÀ‡ gpu.0.gpu‚4.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚116.583' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.14575 gpu.0.smClock‚1245 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ð~Æ¢© ‚(:ø À×çÔÑ™ö‡ disk.md0p1.in‚8.890625 disk.md0.in‚8.890625 disk.md0.out‚ 877.39453125 network.sent‚ 1.8531586e+07( disk./.usagePercent‚83.1844414879093 disk.sda.in‚ 4.40234375# disk.nvme1n1.in‚201288.60546875# proc.memory.rssMB‚ 2951.63671875 network.recv‚ 2.855497e+06 disk.dm-0.in‚531128.94140625 disk.sda1.out‚0 disk.sda2.in‚ 4.40234375 disk.sdb2.in‚OôH/á 4.48828125 disk.sdb.out‚910.27001953125 disk.rbd2.out‚ 2.466796875 cpu‚25.173485420110442! disk.sda2.out‚910.27001953125 disk.rbd0.in‚0.015625" disk.nvme0n1.out‚ 48257.2421875% disk./.usageGB‚2975.0785522460938 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sda.out‚910.27001953125 disk.rbd2.in‚0# disk.nvme1n1.out‚58566.67578125& proc.memory.availableMB‚ 10373.4375 proc.cpu.threads‚17" disk.nvme0n1.in‚329840.3359375! disk.dm-0.out‚106823.91796875 disk.sdb.in‚ 4.48828125! disk.sdb2.out‚910.27001953125 disk.rbd0.out‚ 2.2421875 disk.md0p1.out‚ 877.39453125$ memory_percent‚36.68556213378906* proc.memory.percent‚18.015360832214355ýqñ?2ª - _wandbruntime‚8320  _runtime‚8320ÿü?Ç« Á & lr-AdamW/pg1‚0.0003470993541570001 & lr-AdamW/pg2‚0.0003470993541570001 trainer/global_step‚7199 % _timestamp‚1.7884722678029354e+09 _runtime‚8320.988061671 _step‚587ϳTQ̬ Æ & lr-AdamW/pg1‚0.0003470993541570001 & lr-AdamW/pg2‚0.0003470993541570001 trainer/global_step‚7199 % _timestamp‚1.7884722678029354e+09 _runtime‚8320.988061671 _step‚587Ë°›%2­ - _wandbruntime‚8320  _runtime‚8320‰·$Òî'® èO RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 133247076 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚81462948 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51783418 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 133246366 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 195468609 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 331906428 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚306639479472128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 136437819 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 136437821 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚306639227800576 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚292534290273428 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚306639479472128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚292347905537024 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚306639227800576 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚324238995516416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 _runtime‚8320.989832719 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325045248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚278283452028720 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14251067225792 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚708 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14291340266496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136437821 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _step‚588 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 % _timestamp‚1.7884722678044772e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 136437819 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚710 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 195468609 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 195494953 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 301105152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚278283469068080 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚292347905537024 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 195494953 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚309705636190720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14291573935104% @2ó'¯ íO QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚7199 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14533335385600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚306639227800576 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚324238995516416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 _runtime‚8320.989832719 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325045248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚278283452028720 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14251067225792 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚708 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14291340266496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136437821 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _step‚588 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 % _timestamp‚1.7884722678044772e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 136437819 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚710 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 195468609 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 195494953 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 301105152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚278283469068080 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚292347905537024 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 195494953 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚309705636190720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14291573935104 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 133247076 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚81462948 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51783418 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 133246366 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 195468609 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 331906428 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚306639479472128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 136437819 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 136437821 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚306639227800576 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚292534290273428 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚306639479472128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚292347905537024 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14293304292352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚278320982386144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚292387173571072 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81473752 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚292574025140490 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133264234 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚8322.046778479 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚292573779119838 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14252796733694 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14293304292352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚684 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚306680477863424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 _step‚589 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14293537974272 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 136456094 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚324282355422208 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 331977767 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14293537974272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚81473750 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚324282661579776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 331951395 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚306680729548288 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚292387191574016 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚7199 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 136456094 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 331977767 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 195521671”ײ ÷M UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14252796733694 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14293304292352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚684 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚306680477863424 C9DeviceSÍ»ôÆ&#tatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 _step‚589 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14293537974272 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 136456094 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚324282355422208 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 331977767 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14293537974272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚81473750 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚324282661579776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 331951395 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚306680729548288 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚292387191574016 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚7199 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 136456094 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 331977767 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 195521671 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 136456096 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 195521671 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14293304292352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚278320982386144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚292387173571072 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81473752 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚292574025140490 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133264234 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚8322.046778479 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚292573779119838 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0Í-ø1­2³ - _wandbruntime‚8322  _runtime‚8322¨!ƒ—Y´ Ó 2loss/SRqqt3/assignment_loss‚0.8369022060592636 2loss/SRqqt4/assignment_loss‚1.1116036526201403 /loss/FRt4/detection_loss‚2.9524889156736176 _runtime‚8322.048246396 _step‚590 0loss/FRt4/assignment_loss‚2.4836304434414567 1loss/SRqqt3/detection_loss‚0.6796021562914888 0loss/SRqqt4/detection_loss‚2.317745890687494 epoch‚5 % _timestamp‚1.7884722688635597e+09 0loss/FRt2/assignment_loss‚3.3155191588401793 trainer/global_step‚7199 2loss/SRqqt2/assignment_loss‚1.4609159151358264 1loss/FBt1/assignment_loss‚0.19211687686514742 0loss/FBt4/assignment_loss‚0.1173475726753388 /loss/FRt3/detection_loss‚1.4121957685994475 0loss/SRqqt1/detection_loss‚2.103640507374491 /loss/FRt1/assignment_loss‚2.760820511976878 0loss/FRt3/assignment_loss‚1.0504963021529348 .loss/FBt3/detection_loss‚0.653504216694273 .loss/FBt4/detection_loss‚2.558946330100298 2loss/SRqqt1/assignment_loss‚1.3875286146288826 1loss/SRqqt2/detection_loss‚2.2845111872468675 /loss/FBt2/detection_loss‚2.4470847100019455 1loss/FBt2/assignment_loss‚0.23752734741549658 .loss/FRt1/detection_loss‚2.847278895974159 %loss/total_loss‚40.25494117640679 1loss/FBt3/assignment_loss‚0.04947688644351115 .loss/FRt2/detection_loss‚2.319136048905784 -loss/FBt1/detection_loss‚2.67492106060187­^µ Ø 1loss/FBt1/assignment_loss‚0.19211687686514742 0loss/FBt4/assignment_loss‚0.1173475726753388 /loss/FRt3/detection_loss‚1.4121957685994475 0loss/SRqqt1/detection_loss‚2.103640507374491 /loss/FRt1/assignment_loss‚2.760820511976878 0loss/FRt3/assignment_loss‚1.0504963021529348 .loss/FBt3/detection_loss‚0.653504216694273 .loss/FBt4/detection_loss‚2.558946330100298 2loss/SRqqt1/assignment_loss‚1.3875286146288826 1loss/SRqqt2/detection_loss‚2.2845111872468675 /loss/FBt2/detection_loss‚2.4470847100019455 1loss/FBt2/assignment_loss‚0.23752734741549658 .loss/FRt1/detection_loss‚2.847278895974159 %loss/total_loss‚40.25494117640679 1loss/FBt3/assignment_loss‚0.04947688644351115 .loss/FRt2/detection_loss‚2.319136048905784 -loss/FBt1/detection_loss‚2.67492106060187 2loss/SRqqt3/assignment_loss‚0.8369022060592636 2loss/SRqqt4/assignment_loss‚1.1116036526201403 /loss/FRt4/detection_loss‚2.9524889156736176 _runtime‚8322.048246396 _step‚590 0loss/FRt4/assignment_loss‚2.4836304434414567 1loss/SRqqt3/detection_loss‚0.6796021562914888 0loss/SRqqt4/detection_loss‚2.317745890687494 epoch‚5 % _timestamp‚1.7884722688635597e+09 0loss/FRt2/assignment_loss‚3.3155191588401793 trainer/global_step‚7199 2loss/SRqqt2/assignment_loss‚1.4609159151358264ÎZ]¶¶ ‚(:… Ï×çÔ±¹ã‡ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚139.868' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.967 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼‹ìð€· ‚(:õ Ï×çÔɋሠdisk.sdb.out‚ 911.626953125 disk.sda.out‚ 911.626953125 disk.sda2.in‚ 4.40234375 disk.md0.out‚ 878.69921875 disk.rbd2.out‚ 2.466796875# disk.nvme1n1.in‚201288.71484375 disk.md0p1.in‚8.890625 disk.sdb2.in‚ 4.48828125 disk.sdb2.out‚ 911.626953125% memory_percent‚36.685895919799805% disk./.usageGB‚2975.3285522460938 disk.rbd2.in‚0 disk.sda.in‚ 4.40234375 disk.dm-0.in‚531129.1640625! disk.dm-0.out‚107206.93359375 cpu‚25.173198716440936 network.recv‚ 2.858428e+06 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 2.2421875 disk.sda2.out‚ 911.626953125) proc.memory.availableMB‚ 10373.3828125# proc.memory.rssMB‚ 2951.63671875 proc.cpu.threads‚17 disk.sdb1.out‚0) disk./.usagePercent‚83.19143159254243 disk.sdb.in‚ 4.48828125 disk.sda1.in‚0 disk.md0.in‚8.890625 network.sent‚ 1.8534403e+07 disk.sdb1.in‚0# disk.nvme0n1.in‚329840.44921875# disk.nvme0n1.out‚48640.05078125 disk.sda1.out‚0 disk.md0p1.out‚ 878.69921875* proc.memory.percent‚18.015360832214355" disk.nvme1n1.out‚ 58566.8828125N ‚(:† Þ×çÔ«ÁÒ‡ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚138.302' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.5755 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zG·[„¹ ‚(:ù Þ×çÔÙ‡ù‰# disk.nvme0n1.out‚48769.22265625) disk./.usagePercent‚83.16187787898109 disk.sda.in‚ 4.40234375* proc.memory.percent‚18.015360832214355# disk./.usageGB‚2974.27156829834! disk.sdb2.out‚913.59912109375# disk.nvme1n1.out‚58566.99609375 disk.sdb.in‚ 4.48828125 disk.sdb.out‚913.59912109375# disk.nvme0n1.in‚329840.44921875 disk.sda1.in‚0 cpu‚25.17294102380554 disk.sda.out‚913.59912109375 disk.sda1.out‚0 disk.sda2.in‚ 4.40234375# proc.memory.rssMB‚ 2951.63671875 network.sent‚ 1.8641852e+07 disk.md0p1.out‚ 880.58984375 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 2.2421875 disk.rbd2.in‚0 disk.rbd2.out‚ 2.466796875# disk.nvme1n1.in‚201288.71484375 disk.md0p1.in‚8.890625 disk.sdb2.in‚ 4.48828125 disk.md0.in‚8.890625 disk.md0.out‚ 880.58984375 proc.cpu.threads‚17 network.recv‚ 2.865269e+06 disk.sdb1.out‚0* proc.memory.availableMB‚10373.12890625 disk.dm-0.in‚531129.1640625 disk.dm-0.out‚ 107336.21875! disk.sda2.out‚913.59912109375% memory_percent‚36.687445640563965 disk.sdb1.in‚0f¿û°yº ‚(:î í×çÔé—š‡ disk.sda2.in‚ 4.40234375 disk.sda2.out‚ 915.111328125) disk./.usagePercent‚83.16186411977245 disk.nvme1n1.out‚ 58567.1875 disk.sdb.in‚ 4.48828125 disk.sda.in‚ 4.40234375 disk.sda.out‚ 915.111328125# disk.nvme0n1.in‚329840.44921875% memory_percent‚36.685943603515625% proc.memory.availableMB‚ 10373.375! disk.dm-0.out‚107551.73828125 disk.sdb2.out‚ 915.111328125* proc.memory.percent‚18.015360832214355 disk.md0.out‚ 882.01953125 disk.rbd0.in‚0.015625# disk.nvme1n1.in‚201288.71484375 disk.md0p1.out‚ 882.01953125 disk.rbd2.in‚0 disk.sdb.out‚ 915.111328125 disk.dm-0.in‚531129.1640625 disk.sda1.in‚0 proc.cpu.threads‚17 disk.md0p1.in‚8.890625 network.sent‚ 1.8647672e+07 disk.md0.in‚8.890625 disk.sdb1.out‚0# proc.memory.rssMB‚ 2951.63671875% disk./.usageGB‚2974.2710762023926 disk.rbd0.out‚ 2.2421875 disk.rbd2.out‚ 2.466796875 disk.sdb1.in‚0# disk.nvme0n1.out‚48984.55078125 disk.sda1.out‚0 disk.sdb2.in‚ 4.48828125 cpu‚25.172678501907324 network.recv‚ 2.869603e+06úBd’» ‚(:‡ í×çÔ±»Ï‡ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚141.087' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.27175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07œMFx¼ ‚(:í ü×çÔ‰‰Õ‡ disk.sda.in‚ 4.40234375 disk.sda.out‚ 916.697265625 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.sdb.in‚ 4.48828125 disk.sdb1.out‚0% memory_percent‚36.684465408325195) proc.memory.availableMB‚ 10373.6171875 disk.md0.in‚8.890625 disk.sdb1.in‚0 disk.sda1.out‚0 cpu‚25.172397126852093 proc.cpu.threads‚17 disk.rbd2.out‚ 2.466796875 disk.sda2.in‚ 4.40234375 network.sent‚ 1.8655909e+07 disk.sdb2.in‚ 4.48828125# disk.nvme1n1.in‚201288.71484375" disk.nvme1n1.out‚ 58567.3046875 disk.nvme0n1.out‚ 49048.6875 disk.sda2.out‚ 916.697265625 network.recv‚ 2.874048e+06) disk./.usagePercent‚83.14711915449087$ disk./.usageGB‚2973.743724822998 disk.rbd0.out‚ 2.2421875# proc.memory.rssMB‚ 2951.63671875 disk.md0.out‚ 883.5546875# disk.nvme0n1.in‚329840.44921875 disk.dm-0.in‚531129.1640625 disk.dm-0.out‚107615.9921875 disk.md0p1.in‚8.890625* proc.memory.percent‚18.015360832214355 disk.sda1.in‚0 disk.md0p1.out‚ 883.5546875 disk.sdb2.out‚ 916.697265625 disk.sdb.out‚ 916.697265625»‘½ ‚(:† ü×çÔçš’ˆ gpu.0.gpu‚10.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚102.658' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.6645 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ï)#2¾ - _wandbruntime‚8372  _runtime‚8372š[ÀÚÇ¿ Á & lr-AdamW/pg2‚0.0003412301256733751 trainer/global_step‚7249 % _timestamp‚1.7884723188998327e+09 _runtime‚8372.084917628 _step‚591 & lr-AdamW/pg1‚0.0003412301256733751ÿâ¡ýÌÀ Æ & lr-AdamW/pg2‚0.0003412301256733751 trainer/global_step‚7249 % _timestamp‚1.7884723188998327e+09 _runtime‚8372.084917628 _step‚591 & lr-AdamW/pg1‚0.0003412301256733751ÏÛÊyå2Á - _wandbruntime‚8372  _runtime‚8372JE%Èî' èO QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137351569 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚294312189383680 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14389555973120 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚668 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 134133977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52130929 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 334184476 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚294312207386624 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14389789641728 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚308701745356800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 196832905 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚326423031518720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 196806561 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 334158130 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82003050 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14389789641728 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14349005266300 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 196806561 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚308701745356800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚294312189383680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚308701997028352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚311789858633216 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚311789834693120 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14633172885504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚280159969899920 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14389555973120 _step‚592 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137351569 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306170880 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14632890654720 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52131595 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚308701997028352 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 137351571 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282230784 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚280159986939280 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚294312207386624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 134134645 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚8372.086132544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137351571 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14348776285216 % _timestamp‚1.7884723189009383e+09Ô1ÌUà íO LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚7249 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚294508746185136 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚308701997028352 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 137351571 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282230784 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚280159986939280 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚294312207386624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 134134645 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚8372.086132544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137351571 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14348776285216 % _timestamp‚1.7884723189009383e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137351569 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚294312189383680 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14389555973120 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚668 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 134133977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52130929 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 334184476 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFD ŸÅÅževiceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚294312207386624 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14389789641728 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚308701745356800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 196832905 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚326423031518720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 196806561 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 334158130 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82003050 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14389789641728 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14349005266300 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 196806561 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚308701745356800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚294312189383680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚308701997028352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚311789858633216 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚311789834693120 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14633172885504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚280159969899920 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14389555973120 _step‚592 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137351569 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306170880 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14632890654720 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52131595 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚294351475420672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137369846 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 196859623 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚698 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14635173593088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14350734774202 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚308743247104512 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 334229469 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚294548235031546 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚311831542603264 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 # _timestamp‚1.78847232005404e+09 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14391519998976 trainer/global_step‚7249 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚294351493423616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52138180 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚593 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14391753680896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚326466716196352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚294548481052198 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137369844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52138876 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚280197517296704 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚8373.240771979 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14391753680896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚294351493423616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82013850 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14391519998976 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14634887181312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 334203097 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚326466405844480 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134152030 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82013852 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137369846 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134152030 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82013852 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137369846 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚294351475420672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137369846 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 196859623 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚698 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14635173593088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14350734774202 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚308743247104512 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 334229469 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚294548235031546 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚311831542603264 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 # _timestamp‚1.78847232005404e+09 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14391519998976 trainer/global_step‚7249 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚294351493423616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52138180 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚593 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14391753680896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚326466716196352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚294548481052198 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137369844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52138876 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚280197517296704 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚8373.240771979 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14391753680896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚294351493423616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82013850 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14391519998976 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14634887181312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 334203097 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309Ñã/+b2Ç - _wandbruntime‚8373  _runtime‚8373¨×ÒZÈ Ô /loss/FRt2/assignment_loss‚3.436840018059345 0loss/SRqqt2/detection_loss‚2.350066069400672 0loss/SRqqt4/detection_loss‚2.569099169352959 /loss/FRt3/assignment_loss‚1.345429495761269 2loss/SRqqt3/assignment_loss‚0.8688845091761606 0loss/FBt3/assignment_loss‚0.0908391050881803 2loss/SRqqt4/assignment_loss‚1.8753496435438763 /loss/FRt1/detection_loss‚2.8502526303132374 /loss/FBt4/detection_loss‚2.3490432780236006 %loss/total_loss‚42.42342126769954 2loss/SRqqt1/assignment_loss‚1.4020163127354213 1loss/FBt4/assignment_loss‚0.12379198012057913 1loss/SRqqt1/detection_loss‚1.9945912522929055 1loss/FBt2/assignment_loss‚0.25947148919657126 /loss/FRt3/detection_loss‚1.4099314213534326 .loss/FRt4/detection_loss‚3.205588167621976 /loss/FRt1/assignment_loss‚3.233007329702377 0loss/FRt4/assignment_loss‚3.5310144197373163 /loss/FRt2/detection_loss‚1.9036701224704689 1loss/SRqqt3/detection_loss‚0.7634017772393898 0loss/FBt3/detection_loss‚0.48513461708734257 1loss/SRqqt2/assignment_loss‚1.539846903954943 .loss/FBt2/detection_loss‚2.103385658825145 _runtime‚8373.242262116 _step‚594 0loss/FBt1/assignment_loss‚0.2785413209927686 /loss/FBt1/detection_loss‚2.4542245756496084 epoch‚5 trainer/global_step‚7249 % _timestamp‚1.7884723200573196e+09¨Ù}_É Ù 2loss/SRqqt4/assignment_loss‚1.8753496435438763 /loss/FRt1/detection_loss‚2.8502526303132374 /loss/FBt4/detection_loss‚2.3490432780236006 %loss/total_loss‚42.42342126769954 2loss/SRqqt1/assignment_loss‚1.4020163127354213 1loss/FBt4/assignment_loss‚0.12379198012057913 1loss/SRqqt1/detection_loss‚1.9945912522929055 1loss/FBt2/assignment_loss‚0.25947148919657126 /loss/FRt3/detection_loss‚1.4099314213534326 .loss/FRt4/detection_loss‚3.205588167621976 /loss/FRt1/assignment_loss‚3.233007329702377 0loss/FRt4/assignment_loss‚3.5310144197373163 /loss/FRt2/detection_loss‚1.9036701224704689 1loss/SRqqt3/detection_loss‚0.7634017772393898 0loss/FBt3/detection_loss‚0.48513461708734257 1loss/SRqqt2/assignment_loss‚1.539846903954943 .loss/FBt2/detection_loss‚2.103385658825145 _runtime‚8373.242262116 _step‚594 0loss/FBt1/assignment_loss‚0.2785413209927686 /loss/FBt1/detection_loss‚2.4542245756496084 epoch‚5 trainer/global_step‚7249 % _timestamp‚1.7884723200573196e+09 /loss/FRt2/assignment_loss‚3.436840018059345 0loss/SRqqt2/detection_loss‚2.350066069400672 0loss/SRqqt4/detection_loss‚2.569099169352959 /loss/FRt3/assignment_loss‚1.345429495761269 2loss/SRqqt3/assignment_loss‚0.8688845091761606 0loss/FBt3/assignment_loss‚0.0908391050881803ÒâB²D’Ê ‚(:‡ ‹ØçÔÏÓÕ† gpu.0.gpu‚47.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.357' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.83925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qÈK†Ë ‚(:û ‹ØçÔ¤®œ‡ disk.sdb.in‚ 4.48828125 disk.sdb.out‚918.65478515625 network.sent‚ 1.8658939e+07! disk.dm-0.out‚107661.66015625 disk.sdb1.out‚0 disk.sda.in‚ 4.40234375 cpu‚25.174913917342316 proc.cpu.threads‚17# disk.nvme0n1.in‚329855.34765625 disk.sda1.in‚0# proc.memory.rssMB‚ 3029.88671875) disk./.usagePercent‚83.14387550104851 disk.sdb2.in‚ 4.48828125 disk.md0.in‚8.890625 disk.rbd0.in‚0.015625 disk.rbd2.in‚0# disk.nvme1n1.in‚201313.06640625! disk.sdb2.out‚918.65478515625 disk.dm-0.in‚531168.4140625) proc.memory.availableMB‚ 10294.9453125* proc.memory.percent‚18.492960929870605 disk.sdb1.in‚0 disk.sda.out‚918.65478515625" disk.nvme0n1.out‚ 49093.7265625 disk.sda2.in‚ 4.40234375 disk.md0p1.in‚8.890625 disk.md0p1.out‚ 885.45703125$ memory_percent‚37.16464042663574 disk.rbd0.out‚ 2.2421875# disk.nvme1n1.out‚58567.93359375 disk.sda1.out‚0! disk.sda2.out‚918.65478515625 network.recv‚ 2.877805e+06$ disk./.usageGB‚2973.627716064453 disk.md0.out‚ 885.45703125 disk.rbd2.out‚ 2.466796875Xå ‚‘Ì ‚(:† šØçÔŠ—ȇ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚142.313' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.57825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÕB³pÍ ‚(:å šØçÔ¯Ò⇠disk.rbd0.out‚ 2.2421875 disk.sda1.in‚0 network.recv‚ 2.885075e+06 disk.sdb1.in‚0 disk.dm-0.in‚531168.4140625 disk.md0p1.out‚ 887.03125* proc.memory.availableMB‚10295.20703125# proc.memory.rssMB‚ 3029.88671875 disk.sdb2.in‚ 4.48828125 disk.sdb1.out‚0 disk.sda.out‚ 920.28515625* proc.memory.percent‚18.492960929870605# disk./.usageGB‚2973.65323638916 disk.rbd2.in‚0 disk.sdb.out‚ 920.28515625# disk.nvme0n1.in‚329855.34765625 proc.cpu.threads‚17 network.sent‚ 1.8769144e+07 disk.sdb2.out‚ 920.28515625 disk.rbd2.out‚ 2.466796875" disk.nvme1n1.out‚ 58568.5390625 disk.sda.in‚ 4.40234375 cpu‚25.181387889122337 disk.rbd0.in‚0.015625# disk.nvme1n1.in‚201313.06640625) disk./.usagePercent‚83.14458906000841 disk.dm-0.out‚ 107734.21875 disk.sda1.out‚0 disk.md0p1.in‚8.890625$ memory_percent‚37.16304302215576 disk.md0.out‚ 887.03125 disk.sdb.in‚ 4.48828125" disk.nvme0n1.out‚ 49165.6796875 disk.sda2.in‚ 4.40234375 disk.sda2.out‚ 920.28515625 disk.md0.in‚8.890625¢}çÎ ‚(:÷ ©ØçÔ¥ª±‡# proc.memory.rssMB‚ 3029.88671875 network.recv‚ 2.88944e+06 disk.sda2.in‚ 4.40234375 disk.sdb2.in‚ 4.48828125 disk.rbd2.in‚0# disk.nvme1n1.in‚201313.06640625 disk.sda.out‚921.71728515625 disk.sda1.out‚0 network.sent‚ 1.8772888e+07 disk.sda1.in‚0! disk.sdb2.out‚921.71728515625 disk.md0.in‚8.890625 disk.sdb.out‚921.71728515625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda.in‚ 4.40234375 disk.÷˼қdm-0.in‚531168.4140625% memory_percent‚37.165141105651855% disk./.usageGB‚2973.6863174438477 disk.rbd2.out‚ 2.466796875 disk.dm-0.out‚107762.7421875 cpu‚25.189265068965863 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 2.2421875 disk.nvme1n1.out‚ 58568.78125! disk.sda2.out‚921.71728515625 disk.md0p1.out‚ 888.4140625 disk.sdb.in‚ 4.48828125# disk.nvme0n1.in‚329855.34765625* proc.memory.percent‚18.492960929870605 proc.cpu.threads‚17) disk./.usagePercent‚83.14551402014297 disk.md0p1.in‚8.890625 disk.md0.out‚ 888.4140625" disk.nvme0n1.out‚ 49193.9609375* proc.memory.availableMB‚10294.86328125ÁÛ¿’Ï ‚(:‡ ©ØçÔÀ徇 gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ËÔ8/qÐ ‚(:æ ¸ØçÔŽÛ¬‡ disk.sda1.in‚0 disk.sdb1.in‚0 disk.dm-0.out‚ 107907.25 disk.md0p1.out‚ 890.40234375$ memory_percent‚37.16385364532471 network.recv‚ 2.893479e+06 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 2.2421875# disk.nvme1n1.in‚201313.06640625! disk.nvme1n1.out‚ 58568.921875 disk.sdb2.out‚ 923.76171875 cpu‚25.19663954299049 proc.cpu.threads‚17 disk.sdb1.out‚0# disk.nvme0n1.in‚329855.34765625 disk.sda2.in‚ 4.40234375 disk.sda2.out‚ 923.76171875 disk.sdb2.in‚ 4.48828125* proc.memory.percent‚18.492960929870605) disk./.usagePercent‚83.15674174771729$ disk./.usageGB‚2974.087875366211* proc.memory.availableMB‚10295.07421875 disk.md0.out‚ 890.40234375 disk.rbd2.in‚0 disk.rbd2.out‚ 2.466796875 network.sent‚ 1.8781118e+07 disk.sdb.out‚ 923.76171875! disk.nvme0n1.out‚ 49338.328125 disk.sda1.out‚0 disk.md0p1.in‚8.890625 disk.md0.in‚8.890625# proc.memory.rssMB‚ 3029.88671875 disk.sdb.in‚ 4.48828125 disk.sda.in‚ 4.40234375 disk.sda.out‚ 923.76171875 disk.dm-0.in‚531168.4140625#ïÏÑ ‚(:… ¸ØçÔòÆó‡ gpu.0.gpu‚10.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚139.528' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨ˆ\!2Ò - _wandbruntime‚8432  _runtime‚8432Žü¬T*Ó ÎT ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 560288777 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20711 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3532 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 197401789 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 138196662 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚273 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚295981855609344 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32006 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚14381778724790 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32006 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚310404503876608 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 335577740 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚52457450 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1300 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6236507136 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20711 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚309 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚14422666270208 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚295982012103630 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚281600061558368 $ _timestamp‚1.788472378833382e+09 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚14422842082304 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 193815040 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚82629478 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6458321920 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚310404503876608 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1319 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚582 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚309 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚273 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚14422842082304 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 175812096 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 135086926 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚36 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 347418624 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚14666306157056 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 7134511104 _step‚595 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚389 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚52457837 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚14666463792128 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188859832 _runtime‚8432.018781853 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 335598453 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚295981855609344 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 197381078 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6966900736 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 138196664 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚281600044519008 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚295981837606400 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚14422666270208 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚328152286626816 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 7134511104 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193815040 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚273 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 197381078 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚582 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚310404697691648 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚328152468201984 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚313485980469760 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 197381078 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6966900736 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 138196664 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚281600044519008 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚295981837606400 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚14422666270208 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚328152286626816 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 7134511104 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193815040 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚273 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 197381078 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚582 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚310404697691648 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚328152468201984 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚313485980469760 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 138212044 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚296011903280640 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚296011921283584 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 138212046 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚389 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 7134511104 _runtime‚8433.068403318 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6966900736 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚296011921283584 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 335602030 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3532 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚309 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 138212044 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚296011903280640 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚296011921283584 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 138212046 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚389 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 7134511104 _runtime‚8433.068403318 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6966900736 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚296011921283584 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 335602030 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3532 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚309 B&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2393103390932083 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.48376214504241943 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4144781231880188 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.396244615316391 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2664141356945038 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_1_of_4‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.20099200308322906 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9724462628364563 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5863516926765442 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.020667506381869316 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2183748334646225 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18153512477874756 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22073906660079956 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/0FRt1SRqqt2FBt/event_purity‚0.0416666679084301 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.47900182008743286 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3212968409061432 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07453659176826477 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008671366609632969 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28188011050224304 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.230996236205101 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.292590469121933 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9901286363601685 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.670634925365448 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.575251579284668 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.10633886605501175 particle/accuracy_2_of_2‚1 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4811072051525116 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_4‚0.40429383516311646 (particle/accuracy‚0.8550193309783936 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 trainer/global_step‚7253 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19214925169944763 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10461511462926865 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2207728773355484 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21958020329475403 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.014172335155308247 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 jet/accuracy_4_of_7‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.4000000059604645 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9888389706611633 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_2_of_6‚0.6025640964508057 jet/accuracy_5_of_7‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0833333358168602 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.43294352293014526 8"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.607758641242981 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.18223077058792114 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.44278988242149353 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6441687941551208 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.44186046719551086 particle/accuracy_3_of_3‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 jet/accuracy_5_of_6‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0833333358168602 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4990149140357971 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9902899265289307 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.41577380895614624 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16783185303211212 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6182399988174438 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.37336304783821106 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_4_of_6‚1 particle/accuracy_3_of_7‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ( Purity/2FRt3SRqqt1FBt/FRt_purity‚0.5 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2937699258327484 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.36296287178993225 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.40032678842544556 particle/accuracy_5_of_7‚1 )loss/val_total_loss‚41.27822848029672 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.02425875887274742 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4933333098888397 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6666666865348816 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6139880418777466 *jet/accuracy_1_of_2‚0.7627121210098267 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4270833134651184 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5274645090103149 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3781864047050476 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1983633041381836 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4638157784938812 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4469697177410126 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.40833330154418945 jet/accuracy_1_of_6‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_6_of_6‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.22153504192829132 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2665614187717438 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5455582141876221 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.171875 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1604827642440796 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.37497907876968384 6 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.984706461429596 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.588665246963501 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5596529841423035 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 particle/accuracy_2_of_7‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5106034278869629 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1315789520740509 8"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.550000011920929 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 6 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.170183002948761 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9894508123397827 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888889253139496 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4063253700733185 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5393518805503845 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.48188021779060364 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.42980456352233887 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.48758745193481445 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9852729439735413 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5162898898124695 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333134651184 *jet/accuracy_3_of_6‚0.1666666567325592 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2117816060781479 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.2236841917037964 *jet/accuracy_1_of_3‚0.8789499402046204 particle/accuracy_3_of_5‚1 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1680489480495453 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5039106607437134 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9951314330101013 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7348485589027405 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.33361494541168213 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.22902297973632812 particle/accuracy_5_of_5‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2564716339111328 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.3137977421283722 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5039106607437134 ,jet/accuracy_4_of_5‚0.004480286967009306 +particle/specificity‚0.9506926536560059 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.15359504520893097 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4130997955799103 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.995|?wºK17800426483154 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6578947305679321 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 +jet/accuracy_1_of_1‚0.49190428853034973 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 particle/accuracy_5_of_6‚1 particle/accuracy_4_of_7‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6006410121917725 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01659136638045311 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.39933866262435913 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5171370506286621 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5530056357383728 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4926108121871948 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_7_of_7‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.15395525097846985 particle/accuracy_2_of_4‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09038350731134415 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.02886904776096344 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5520833134651184 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9904184341430664 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 'particle/f_score‚0.4211549162864685 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 _step‚597 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9722222685813904 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2023809552192688 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5083096623420715 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3728632926940918 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.29041963815689087 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1560283601284027 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5162999629974365 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9062500596046448 jet/accuracy_2_of_7‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5725308656692505 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.43505746126174927 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6470588445663452 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4903853237628937 particle/accuracy_1_of_7‚1 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9169909954071045 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ;"Purity/2FRt1SRqqt0FBt/event_purity‚0.002910052891820669 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5230178236961365 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9884465336799622 particle/accuracy_4_of_5‚1 jet/accuracy_1_of_7‚0.5 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4022849500179291 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 particle/accuracy_2_of_5‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5313079357147217 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5370690226554871 +particle/sensitivity‚0.3380327522754669 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3950617015361786 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2985301613807678 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.025673815980553627 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.009410430677235126 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5459164381027222 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5562916398048401 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.01632104627788067 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5804126858711243 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5024906992912292 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.014998053200542927 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.551196813583374 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9979310631752014 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3961873948574066 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6946497559547424 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 *"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9894528388977051 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12855882942676544 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4057057499885559 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.08174363523721695 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.43115389347076416 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1878444105386734 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 particle/accuracy_1_of_2‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2708333432674408 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_4‚1 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.8571428656578064 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.65625 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007353168446570635 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.28333333134651184 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9917122721672058 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24790173768997192 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1983633041381836 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5068106055259705 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.15231481194496155 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5527430176734924 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.12878787517547607 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5536355376243591 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735632538795471 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 % _timestamp‚1.7884723947187467e+09 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06612299382686615 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4441179633140564 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.032271210104227066 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5260006189346313 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.08810803294181824 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5357142686843872 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974507093429565 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5060429573059082 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_6‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5881344676017761 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6398919224739075 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.7017543911933899 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1785714328289032 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4114113450050354 particle/accuracy_2_of_3‚1 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0022675737272948027 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9906985759735107 6 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.228202685713768 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9874297976493835 +"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.75 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.18191967904567719 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02840152196586132 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17575135827064514 +jet/accuracy_2_of_2‚0.11590491980314255 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22530865669250488 :"Purity/1FRt2SRqqt*FBt/event_purity‚0.00186741363722831 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9857269525527954 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9896776676177979 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5698562264442444 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_6‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.038987334817647934 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.13333334028720856 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22426174581050873 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.16097517311573029 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5172471404075623 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998404324054718 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.018175283446907997 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5481242537498474 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8296821713447571 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0025767881888896227 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.877193033695221 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 _runtime‚8447.906089411 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.517247200012207 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2708333432674408 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983899593353271 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5196768641471863 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9054374098777771 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 jet/accuracy_5_of_5‚0 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24403592944145203 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 +jet/accuracy_4_of_6‚0.01923076994717121 particle/accuracy_6_of_7‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10818343609571457 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9821364283561707 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.26643702387809753 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4925045073032379 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ©†u84Ü ‹¦ A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2708333432674408 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983899593353271 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5196768641471863 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9054374098777771 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 jet/accuracy_5_of_5‚0 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24403592944145203 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 +jet/accuracy_4_of_6‚0.01923076994717121 particle/accuracy_6_of_7‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10818343609571457 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9821364283561707 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.26643702387809753 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4925045073032379 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9670635461807251 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.14403292536735535 particle/accuracy_6_of_6‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1865079253911972 jet/accuracy_3_of_7‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18557260930538177 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5170348882675171 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9853801131248474 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9907649755477905 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.15325669944286346 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.004471190273761749 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 epoch‚5 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5025423765182495 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18877601623535156 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.23137609660625458 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4391804337501526 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.19711163640022278 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.920634925365448 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.28378379344940186 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.36249998211860657 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5961806178092957 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_6_of_7‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.22719475626945496 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5254310369491577 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.14068585634231567 *jet/accuracy_1_of_5‚0.9499423503875732 particle/accuracy_1_of_3‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2393103390932083 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.48376214504241943 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4144781231880188 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.396244615316391 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2664141356945038 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_1_of_4‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.20099200308322906 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9724462628364563 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5863516926765442 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.020667506381869316 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2183748334646225 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18153512477874756 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22073906660079956 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/0FRt1SRqqt2FBt/event_purity‚0.0416666679084301 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.47900182008743286 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3212968409061432 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07453659176826477 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008671366609632969 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28188011050224304 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.230996236205101 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.292590469121933 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9901286363601685 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.670634925365448 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.575251579284668 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.10633886605501175 particle/accuracy_2_of_2‚1 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4811072051525116 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_4‚0.40429383516311646 (particle/accuracy‚0.8550193309783936 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 trainer/global_step‚7253 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19214925169944763 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10461511462926865 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2207728773355484 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21958020329475403 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.014172335155308247 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 jet/accuracy_4_of_7‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.4000000059604645 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9888389706611633 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_2_of_6‚0.6025640964508057 jet/accuracy_5_of_7‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0833333358168602 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.43294352293014526 8"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.607758641242981 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.18223077058792114 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.44278988242149353 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6441687941551208 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.44186046719551086 particle/accuracy_3_of_3‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 jet/accuracy_5_of_6‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0833333358168602 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4990149140357971 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9902899265289307 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.41577380895614624 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16783185303211212 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6182399988174438 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.37336304783821106 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_4_of_6‚1 particle/accuracy_3_of_7‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ( Purity/2FRt3SRqqt1FBt/FRt_purity‚0.5 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2937699258327484 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.36296287178993225 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.40032678842544556 particle/accuracy_5_of_7‚1 )loss/val_total_loss‚41.27822848029672 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.02425875887274742 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4933333098888397 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6666666865348816 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6139880418777466 *jet/accuracy_1_of_2‚0.7627121210098267 9"Purity/1FRt3SRqqt1FBt/SÜ*¡Ú^Rqqt_purity‚0.4270833134651184 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5274645090103149 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3781864047050476 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1983633041381836 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4638157784938812 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4469697177410126 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.40833330154418945 jet/accuracy_1_of_6‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_6_of_6‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.22153504192829132 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2665614187717438 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5455582141876221 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.171875 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1604827642440796 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.37497907876968384 6 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.984706461429596 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.588665246963501 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5596529841423035 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 particle/accuracy_2_of_7‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5106034278869629 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1315789520740509 8"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.550000011920929 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 6 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.170183002948761 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9894508123397827 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888889253139496 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4063253700733185 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5393518805503845 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.48188021779060364 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.42980456352233887 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.48758745193481445 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9852729439735413 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5162898898124695 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333134651184 *jet/accuracy_3_of_6‚0.1666666567325592 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2117816060781479 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.2236841917037964 *jet/accuracy_1_of_3‚0.8789499402046204 particle/accuracy_3_of_5‚1 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1680489480495453 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5039106607437134 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9951314330101013 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7348485589027405 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.33361494541168213 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.22902297973632812 particle/accuracy_5_of_5‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2564716339111328 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.3137977421283722 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5039106607437134 ,jet/accuracy_4_of_5‚0.004480286967009306 +particle/specificity‚0.9506926536560059 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.15359504520893097 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4130997955799103 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9917800426483154 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6578947305679321 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 +jet/accuracy_1_of_1‚0.49190428853034973 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 particle/accuracy_5_of_6‚1 particle/accuracy_4_of_7‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6006410121917725 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01659136638045311 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.39933866262435913 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5171370506286621 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5530056357383728 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4926108121871948 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_7_of_7‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.15395525097846985 particle/accuracy_2_of_4‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09038350731134415 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.02886904776096344 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5520833134651184 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9904184341430664 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 'particle/f_score‚0.4211549162864685 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 _step‚597 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9722222685813904 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2023809552192688 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5083096623420715 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3728632926940918 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.29041963815689087 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1560283601284027 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5162999629974365 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9062500596046448 jet/accuracy_2_of_7‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5725308656692505 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.43505746126174927 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6470588445663452 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4903853237628937 particle/accuracy_1_of_7‚1 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9169909954071045 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ;"Purity/2FRt1SRqqt0FBt/event_purity‚0.002910052891820669 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5230178236961365 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9884465336799622 particle/accuracy_4_of_5‚1 jet/accuracy_1_of_7‚0.5 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4022849500179291 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 particle/accuracy_2_of_5‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5313079357147217 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5370690226554871 +particle/sensitivity‚0.3380327522754669 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3950617015361786 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2985301613807678 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.025673815980553627 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.009410430677235126 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5459164381027222 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5562916398048401 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.01632104627788067 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5804126858711243 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5024906992912292 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.014998053200542927 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.551196813583374 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9979310631752014 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3961873948574066 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6946497559547424 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 *"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9894528388977051 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12855882942676544 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4057057499885559 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.08174363523721695 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.43115389347076416 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1878444105386734 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 particle/accuracy_1_of_2‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2708333432674408 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_4‚1 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.8571428656578064 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.65625 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007353168446570635 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.28333333134651184 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9917122721672058 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24790173768997192 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1983633041381836 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5068106055259705 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.15231481194496155 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5527430176734924 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.12878787517547607 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5536355376243591 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735632538795471 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 % _timestamp‚1.7884723947187467e+09 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06612299382686615 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4441179633140564 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.032271210104227066 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5260006189346313 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.08810803294181824 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5357142686843872 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974507093429565 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5060429573059082 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_6‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5881344676017761 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6398919224739075 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.7017543911933899 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1785714328289032 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4114113450050354 particle/accuracy_2_of_3‚1 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0022675737272948027 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9906985759735107 6 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.228202685713768 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9874297976493835 +"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.75 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.18191967904567719 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02840152196586132 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17575135827064514 +jet/accuracy_2_of_2‚0.11590491980314255 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22530865669250488 :"Purity/1FRt2SRqqt*FBt/event_purity‚0.00186741363722831 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9857269525527954 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9896776676177979 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5698562264442444 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_6‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.038987334817647934 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.13333334028720856 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22426174581050873 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.16097517311573029 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5172471404075623 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998404324054718 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.018175283446907997 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5481242537498474 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8296821713447571 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0025767881888896227 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.877193033695221 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 _runtime‚8447.906089411 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.517247200012207 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0ÕWùOÝ ‚(:… ÖØçÔûÍц gpu.0.gpu‚44.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚130.564' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.641 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lÙ0~Þ ‚(:ó ÖØçÔÒÙ€‡ disk.sdb1.out‚0 disk.rbd0.in‚0.015625 disk.sdb1.in‚0 disk.sda1.in‚0 disk.md0p1.in‚8.890625 disk.rbd2.out‚ 2.466796875 network.sent‚ 1.8892048e+07! disk.sdb2.out‚927.20654296875 disk.md0.in‚8.890625 disk.rbd0.out‚ 2.2421875 disk.sdb.in‚ 4.48828125 disk.sdb.out‚927.20654296875 disk.sda.in‚ 4.40234375 disk.nvme0n1.out‚ 49663.90625 disk.dm-0.in‚531168.4140625! disk.sda2.out‚927.20654296875 disk.md0p1.out‚ 893.7265625# disk.nvme1n1.out‚58910.58984375 disk.sda1.out‚0 disk.sda2.in‚ 4.40234375 disk.sdb2.in‚ 4.48828125 disk.rbd2.in‚0( proc.memory.availableMB‚ 10292.265625) proc.memory.percent‚18.50724220275879 cpu‚25.203688454730134( disk./.usagePercent‚83.1637068937859 disk.sda.out‚927.20654296875# disk.nvme0n1.in‚329855.34765625 disk.md0.out‚ 893.7265625" proc.memory.rssMB‚ 3032.2265625 proc.cpu.threads‚17 network.recv‚ 2.904703e+06$ disk./.usageGB‚2974.336982727051# disk.nvme1n1.in‚201313.06640625! disk.dm-0.out‚108574.49609375$ memory_percent‚37.18099594116211àLúx‘ß ‚(:† åØçÔÄÞ‡ gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚113.798' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.4495 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xþ¼yà ‚(:î åØçÔºº³ˆ network.sent‚ 1.9017225e+07 disk.sda2.out‚ 928.884765625% memory_percent‚37.183308601379395 disk.rbd2.in‚0! disk.nvme0n1.out‚ 49819.140625 disk.dm-0.out‚108730.1328125 disk.sda1.in‚0 disk.md0.in‚8.890625 disk.rbd2.out‚ 2.466796875 disk.sdb1.in‚0 disk.dm-0.in‚531168.4140625 disk.md0.out‚ 895.33984375" proc.memory.rssMB‚ 3032.2265625 cpu‚25.203262003744747 disk.rbd0.in‚0.015625# disk.nvme1n1.in‚201313.06640625" disk.nvme1n1.out‚ 58910.9921875 disk.sdb.out‚ 928.884765625 disk.sda.in‚ 4.40234375 disk.sda.out‚ 928.884765625# disk.nvme0n1.in‚329855.34765625 disk.sdb2.in‚ 4.48828125 network.recv‚ 2.9125e+06 disk.sdb.in‚ 4.48828125 disk.sda1.out‚0 disk.md0p1.in‚8.890625 disk.rbd0.out‚ 2.2421875 disk.sdb1.out‚0 disk.md0p1.out‚ 895.33984375) disk./.usagePercent‚83.16370860035443$ disk./.usageGB‚2974.337043762207 disk.sdb2.out‚ 928.884765625) proc.memory.percent‚18.50724220275879 proc.cpu.threads‚17 disk.sda2.in‚ 4.40234375* proc.memory.availableMB‚10291.88671875pŽ¤ºá ‚(:… ôØçÔ–ÙÞ… gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚87.451' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚21.86275 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨y÷É„â ‚(:ù ôØçÔÃúí† disk.sda2.in‚ 4.40234375 disk.rbd2.out‚ 2.466796875 disk.sdb.in‚ 4.48828125 disk.sdb1.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚83.14895734210145! disk.sda2.out‚930.72412109375 disk.sdb2.in‚ 4.48828125 cpu‚25.202855108914637 network.recv‚ 2.916696e+06 disk.sda1.out‚0 disk.md0.out‚ 897.10546875% memory_percent‚37.181758880615234( proc.memory.availableMB‚ 10292.140625" proc.memory.rssMB‚ 3032.2265625$ disk./.usageGB‚2973.809467315674 disk.dm-0.in‚531168.4140625 disk.dm-0.out‚108773.7890625! disk.sdb2.out‚930.72412109375 disk.md0.in‚8.890625 disk.sdb1.in‚0# disk.nvme0n1.in‚329855.34765625# disk.nvme0n1.out‚49862.69140625 disk.rbd2.in‚0# disk.nvme1n1.out‚58911.09765625 disk.sda.in‚ 4.40234375 disk.sda.out‚930.72412109375 disk.md0p1.out‚ 897.10546875 disk.rbd0.in‚0.015625) proc.memory.percent‚18.50724220275879 network.sent‚ 1.9021024e+07 disk.sda1.in‚0 disk.md0p1.in‚8.890625 disk.rbd0.out‚ 2.2421875# disk.nvme1n1.in‚201313.06640625 disk.sdb.out‚930.72412109375 gÌB2ã - _wandbruntime‚8499  _runtime‚8499oŠÉä à _step‚598 ' lr-AdamW/pg1‚0.00033537569269671147 ' lr-AdamW/pg2‚0.00033537569269671147 trainer/global_step‚7299 % _timestamp‚1.7884724465587995e+09 _runtime‚8499.743765378çÊ ãÎå È ' lr-AdamW/pg1‚0.00033537569269671147 ' lr-AdamW/pg2‚0.00033537569269671147 trainer/global_step‚7299 % _timestamp‚1.7884724465587995e+09 _runtime‚8499.743765378 _step‚598Öþg2æ - _wandbruntime‚8499  _runtime‚8499ÕE‰« ç èO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚283646745697936 % _timestamp‚1.7884724465595999e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 139234387 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 198733998 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚298170646713344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚298123226223194 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚312688270566912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚315786823672320 trainer/global_step‚7299 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14517641856512 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚298170628710400 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚312688270566912 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚298170646vÖ+C713344 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 198733998 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 139234385 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚315786847612416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14517875525120 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚312688522238464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚283646762737296 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _runtime‚8499.744726903 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 139234385 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14517641856512 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 337968385 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚330549681300992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚330549995860480 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 337942039 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚690 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚298170628710400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52851656 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 198707654 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 136130772 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14517875525120 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14762857628672 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14476709506342 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚312688522238464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 139234387 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52850968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14763148248064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚315786847612416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14517875525120 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚312688522238464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚283646762737296 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _runtime‚8499.744726903 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 139234385 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14517641856512 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 337968385 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚330549681300992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚330549995860480 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 337942039 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚690 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚298170628710400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52851656 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 198707654 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 136130772 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14517875525120 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14762857628672 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14476709506342 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚312688522238464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 139234387 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52850968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14763148248064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚283646745697936 % _timestamp‚1.7884724465595999e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 139234387 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 198733998 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚298170646713344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚298123226223194 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚312688270566912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚315786823672320 trainer/global_step‚7299 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14517641856512 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚298170628710400 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚312688270566912 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚298170646713344 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 198733998 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 139234385 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14519839564288 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚312729772314624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139252662 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚298209932750336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14765142598656 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚298162715069604 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚298209914747392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 % _timestamp‚1.7884724476451364e+09 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14519605882368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚658 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 198734346 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚330593347052032 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚315828531582464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚600 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚312729772314624 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚315828507642368 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 337987006 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136147430 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14519605882368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 338013378 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 327129088 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚283684293094720 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83289918 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14478667995536 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 139252660 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 338013378 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚330593674181120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14519839564288 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚312729520629760 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14764839409664 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚283684276055360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚298209932750336 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52857514 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624°ƒ }ë ÷M _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚315828507642368 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 337987006 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136147430 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14519605882368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 338013378 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 327129088 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚283684293094720 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83289918 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14478667995536 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 139252660 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 338013378 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚330593674181120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14519839564288 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚312729520629760 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14764839409664 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚283684276055360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚298209932750336 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52857514 QCDeviceStatsMonitor.on_train_batch_end/inactive_sp¿…Õ€lit_bytes.all.peak‚ 347418624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 136148090 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 303188992 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚298162961090256 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14519839564288 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚312729772314624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139252662 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚298209932750336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14765142598656 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚298162715069604 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚298209914747392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 % _timestamp‚1.7884724476451364e+09 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14519605882368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚658 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 198734346 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚330593347052032 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚315828531582464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚600 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚312729772314624 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920Øï¡!2ì - _wandbruntime‚8500  _runtime‚8500–Ò´ñ[í Õ 1loss/SRqqt1/assignment_loss‚1.387944745974472 2loss/SRqqt4/assignment_loss‚1.6263401905695598 1loss/FBt2/assignment_loss‚0.14105992265448605 1loss/FBt4/assignment_loss‚0.14661173056808033 /loss/FBt3/detection_loss‚0.5403823334843654 0loss/FRt2/assignment_loss‚3.6231897115707397 /loss/FRt4/detection_loss‚3.1896711795226387 1loss/SRqqt4/detection_loss‚2.5044839713308544 epoch‚6 0loss/FRt3/assignment_loss‚1.2619945001602173 1loss/SRqqt3/assignment_loss‚0.732005730408587 1loss/FBt3/assignment_loss‚0.03471098579189464 1loss/SRqqt1/detection_loss‚1.9913682104685368 /loss/SRqqt2/detection_loss‚2.33784310279354 /loss/FBt1/detection_loss‚2.2826634486516317 0loss/FRt4/assignment_loss‚3.1885117914365684 1loss/SRqqt2/assignment_loss‚1.583942913240002 /loss/FRt3/detection_loss‚1.3294555680205424 1loss/SRqqt3/detection_loss‚0.9576987066943394 /loss/FBt4/detection_loss‚2.4940787966434774 %loss/total_loss‚41.59358319893185 .loss/FRt1/detection_loss‚2.728909367864782 .loss/FRt2/detection_loss‚2.036820746569096 trainer/global_step‚7299 _runtime‚8500.831888819 _step‚601 % _timestamp‚1.7884724476471572e+09 1loss/FBt1/assignment_loss‚0.11004392430169598 .loss/FBt2/detection_loss‚2.198265242576599 0loss/FRt1/assignment_loss‚3.1655863776351465Lû`î Ú 0loss/FRt3/assignment_loss‚1.2619945001602173 1loss/SRqqt3/assignment_loss‚0.732005730408587 1loss/FBt3/assignment_loss‚0.03471098579189464 1loss/SRqqt1/detection_loss‚1.9913682104685368 /loss/SRqqt2/detection_loss‚2.33784310279354 /loss/FBt1/detection_loss‚2.2826634486516317 0loss/FRt4/assignment_loss‚3.1885117914365684 1loss/SRqqt2/assignment_loss‚1.583942913240002 /loss/FRt3/detection_loss‚1.3294555680205424 1loss/SRqqt3/detection_loss‚0.9576987066943394 /loss/FBt4/detection_loss‚2.4940787966434774 %loss/total_loss‚41.59358319893185 .loss/FRt1/detection_loss‚2.728909367864782 .loss/FRt2/detection_loss‚2.036820746569096 trainer/global_step‚7299 _runtime‚8500.831888819 _step‚601 % _timestamp‚1.7884724476471572e+09 1loss/FBt1/assignment_loss‚0.11004392430169598 .loss/FBt2/detection_loss‚2.198265242576599 0loss/FRt1/assignment_loss‚3.1655863776351465 1loss/SRqqt1/assignment_loss‚1.387944745974472 2loss/SRqqt4/assignment_loss‚1.6263401905695598 1loss/FBt2/assignment_loss‚0.14105992265448605 1loss/FBt4/assignment_loss‚0.14661173056808033 /loss/FBt3/detection_loss‚0.5403823334843654 0loss/FRt2/assignment_loss‚3.6231897115707397 /loss/FRt4/detection_loss‚3.1896711795226387 1loss/SRqqt4/detection_loss‚2.5044839713308544 epoch‚6Ù _Ð4ï ‚(:„ ƒÙçÔÅß›ˆ gpu.0.gpu‚8.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚134.908' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.727 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„îÓŠ|ð ‚(:ñ ƒÙçÔÕÜìˆ disk.dm-0.in‚531168.4140625 disk.sda1.out‚0 disk.sdb2.in‚ 4.48828125% memory_percent‚37.181997299194336 cpu‚25.20245714480364 disk.md0.in‚8.890625) proc.memory.availableMB‚ 10292.1015625 disk.sdb1.out‚0# disk.nvme0n1.in‚329855.34765625! disk.dm-0.out‚108780.77734375 disk.sda2.out‚932.2138671875 disk.md0p1.in‚8.890625% disk./.usageGB‚2973.7711448669434 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 2.2421875 disk.sdb.out‚932.2138671875 disk.md0p1.out‚ 898.5390625) disk./.usagePercent‚83.14788583039577 disk.rbd2.out‚ 2.466796875 disk.nvme1n1.out‚ 58911.34375 proc.cpu.threads‚17# disk.nvme0n1.out‚49869.43359375 disk.sda1.in‚0" proc.memory.rssMB‚ 3032.2265625 disk.md0.out‚ 898.5390625 disk.sdb1.in‚0 disk.sda2.in‚ 4.40234375 disk.sdb2.out‚932.2138671875) proc.memory.percent‚18.50724220275879 network.sent‚ 1.902629e+07 network.recv‚ 2.920439e+06 disk.rbd2.in‚0# disk.nvme1n1.in‚201313.06640625 disk.sdb.in‚ 4.48828125 disk.sda.in‚ 4.40234375 disk.sda.out‚932.2138671875‡ÍÉ6ñ ‚(:… ’ÙçÔ‚à†† gpu.0.gpu‚11.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚101.376' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.344 gpu.0.smClock‚1215 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ+„ò ‚(:ù ’ÙçÔøúþ‡ disk.sdb1.out‚0 network.recv‚ 2.927928e+06# disk.nvme1n1.in‚201313.06640625 disk.md0.out‚ 900.23828125 proc.cpu.threads‚17% disk./.usageGB‚2973.8408432006836 disk.sda.in‚ 4.40234375 disk.sda.out‚933.98681640625 disk.md0p1.in‚8.890625 disk.sdb2.in‚ 4.48828125 disk.rbd0.out‚ 2.2421875$ memory_percent‚37.18218803405762) proc.memory.availableMB‚ 10292.0703125) disk./.usagePercent‚83.14983462497815 disk.rbd2.in‚0# disk.nvme1n1.out‚58911.44921875 disk.sdb.out‚933.98681640625# disk.nvme0n1.out‚49941.37890625 disk.dm-0.in‚531168.4140625) proc.memory.percent‚18.50724220275879 network.sent‚ 1.9136644e+07 disk.sdb1.in‚0# disk.nvme0n1.in‚329855.34765625 disk.dm-0.out‚ 108852.828125 disk.sda1.in‚0! disk.sda2.out‚933.98681640625 disk.md0p1.out‚ 900.23828125 disk.rbd2.out‚ 2.466796875 disk.sdb.in‚ 4.48828125! disk.sdb2.out‚933.98681640625 disk.sda1.out‚0 disk.md0.in‚8.890625 disk.rbd0.in‚0.015625 disk.sda2.in‚ 4.40234375" proc.memory.rssMB‚ 3032.2265625 cpu‚25.202107509637752 üAèó ‚(:… ¡ÙçÔô«„‡ gpu.0.gpu‚10.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚32.0 gpu.0.powerWatts‚88.678' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚22.1695 gpu.0.smClock‚1185 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹ÒG~ô ‚(:ó ¡ÙçԭΈ disk.sdb1.in‚0 disk.dm-0.in‚531168.4140625 disk.md0p1.out‚ 901.984375! disk.sdb2.out‚935.79345703125 disk.md0.in‚8.890625$ memory_percent‚37.18142509460449 cpu‚25.20181049890542 disk.sda1.in‚0$ disk./.usageGB‚2974.030632019043# disk.nvme0n1.in‚329855.34765625 disk.rbd0.out‚ 2.2421875 disk.rbd2.out‚ 2.466796875# disk.nvme1n1.in‚201313.06640625) proc.memory.availableMB‚ 10292.1953125" proc.memory.rssMB‚ 3032.2265625 network.recv‚ 2.932357e+06 disk.sda1.out‚0 disk.md0.out‚ 901.984375 network.sent‚ 1.9142569e+07# disk.nvme0n1.out‚50041.68359375! disk.dm-0.out‚108953.35546875 disk.sdb.in‚ 4.48828125 disk.md0p1.in‚8.890625 disk.sdb.out‚935.79345703125 disk.sda.in‚ 4.40234375 disk.sda.out‚935.79345703125 disk.sda2.in‚ 4.40234375) disk./.usagePercent‚83.15514119977227! disk.sda2.out‚935.79345703125 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.sdb2.in‚ 4.48828125! disk.nvme1n1.out‚ 58911.671875) proc.memory.percent‚18.50724220275879 proc.cpu.threads‚17ÝÉ”‘õ ‚(:† °ÙçÔ›±å‡ gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚32.0 gpu.0.powerWatts‚87.451' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚21.86275 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ξ!{ö ‚(:ð °ÙçÔÓ£ˆ disk.sdb2.out‚937.4267578125 cpu‚25.20154489001424 disk.md0.out‚ 903.5546875) disk./.usagePercent‚83.16388928329583 disk.sdb.in‚ 4.48828125 disk.sdb1.out‚0 disk.sda2.in‚ 4.40234375) proc.memory.availableMB‚ 10291.9453125 disk.rbd0.out‚ 2.2421875$ disk./.usageGB‚2974.343505859375# disk.nvme1n1.in‚201313.06640625 disk.sda1.out‚0 disk.sda2.out‚937.4267578125 disk.sdb2.in‚ 4.48828125 network.sent‚ 1.914641e+07 disk.dm-0.in‚531168.4140625! disk.dm-0.out‚109077.16796875 disk.sda1.in‚0$ memory_percent‚37.18295097351074 proc.cpu.threads‚17 network.recv‚ 2.93683e+06" disk.nvme1n1.out‚ 58911.7890625# disk.nvme0n1.in‚329855.34765625# disk.nvme0n1.out‚50165.37890625 disk.md0p1.out‚ 903.5546875 disk.md0.in‚8.890625) proc.memory.percent‚18.50724220275879 disk.sda.in‚ 4.40234375" proc.memory.rssMB‚ 3032.2265625 disk.sdb.out‚937.4267578125 disk.md0p1.in‚8.890625 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.rbd2.out‚ 2.466796875 disk.sdb1.in‚0 disk.sda.out‚937.4267578125hÎçÄ2÷ - _wandbruntime‚8561  _runtime‚8561¡—'ŽÉø à _runtime‚8561.098189899 _step‚602 ' lr-AdamW/pg1‚0.00032953727528206665 ' lr-AdamW/pg2‚0.00032953727528206665 trainer/global_step‚7349 % _timestamp‚1.7884725079131124e+09øפÎù È ' lr-AdamW/pg1‚0.00032953727528206665 ' lr-AdamW/pg2‚0.00032953727528206665 trainer/global_step‚7349 % _timestamp‚1.7884725079131124e+09 _runtime‚8561.098189899 _step‚602Úê:«º2ú - _wandbruntime‚8561  _runtime‚8561n@›8í'û çO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚314750788123136 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚690 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83819216 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚285523280608496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 $ _timestamp‚1.788472507914395e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14615857563136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 200045606 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14574418565766 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140148135 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚314750788123136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14615857563136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚8561.099923117 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚332734025105920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚317871046114816 trainer/global_step‚7349 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 200045606 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14862978991104 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚300097928155346 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚300134948562944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚300134930560000 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 340220087 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14616091231744 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚314751039794688 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚314751039794688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 140148137 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 137019612 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚285523263569136 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53200398 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚332733710546432 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚300097682134902 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14616091231744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83819214 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14862688371712 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _step‚603|Ayðô ü ìO O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14862688371712 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _step‚603 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚314750788123136 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚690 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83819216 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚285523280608496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 $ _timestamp‚1.788472507914395e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14615857563136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 200045606 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14574418565766 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140148135 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚314750788123136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14615857563136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚8561.099923117 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚332734025105920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚317871046114816 trainer/global_step‚7349 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 200045606 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14862978991104 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚300097928155346 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚300134948562944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚300134930560000 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 340220087 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14616091231744 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚314751039794688 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚314751039794688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 140148137 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 137019612 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚285523263569136 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53200398 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚332733710546432 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚300097682134902 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14616091231744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83819214Ûaça–2ý - _wandbruntime‚8562  _runtime‚8562ËÄnœø&þ òM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _runtime‚8562.313197433 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚300137170981312 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140166410 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53207670 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 trainer/global_step‚7349 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280120320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83830018 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 340265080 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚300174216596992 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137037043 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 140166412 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚645 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚314792289870848 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 340238708 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚332777715059200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14618055270912 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 340265080 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 200098668 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚643 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚332777410998784 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140166410 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚317912706144768 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14864704854016 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140166412 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚314792038185984 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 200072298 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83830016 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14576377054752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53207027 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _step‚604 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚317912730084864 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚285560793926560 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚300174234599936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚300174234599936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14576606036044 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚285560810965920 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 340238708 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 % _timestamp‚1.7884725091270742e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚300174216596992 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14617821588992ç&Ôý&ÿ ÷M OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140166412 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚314792038185984 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 200072298 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83830016 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14576377054752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53207027 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _step‚604 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚317912730084864 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚285560793926560 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚300174234599936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚300174234599936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14576606036044 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚285560810965920 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 340238708 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 % _timestamp‚1.7884725091270742e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚300174216596992 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14617821588992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _runtime‚8562.313197433 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚300137170981312 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140166410 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53207670 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 trainer/global_step‚7349 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280120320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83830018 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 340265080 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚300174216596992 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137037043 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 140166412 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚645 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚314792289870848 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 340238708 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚332777715059200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14618055270912 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 340265080 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 200098668 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚643 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚332777410998784 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140166410 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚317912706144768 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14864704854016 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292ÜÎæA¨2€!- _wandbruntime‚8562  _runtime‚8562%/Dq]!× 2loss/SRqqt4/assignment_loss‚2.2984670859116774 0loss/SRqqt2/detection_loss‚2.592668411321938 1loss/SRqqt3/detection_loss‚0.7537170914425091 2loss/SRqqt2/assignment_loss‚1.6663930359645747 1loss/FBt1/assignment_loss‚0.14224880363872217 .loss/FRt2/detection_loss‚2.029809320369887 /loss/FBt4/detection_loss‚2.2274883730070933 % _timestamp‚1.7884725091300516e+09 _runtime‚8562.315005251 0loss/FBt2/assignment_loss‚0.3174175227184375 /loss/FRt3/detection_loss‚1.3364764419139217 /loss/SRqqt4/detection_loss‚2.49925775711353 .loss/FBt2/detection_loss‚1.940285180744372 trainer/global_step‚7349 2loss/SRqqt3/assignment_loss‚1.0795084808487445 1loss/FBt4/assignment_loss‚0.21581304851653219 0loss/FRt1/assignment_loss‚2.9703765398747213 /loss/FRt1/detection_loss‚2.6365655402879455 .loss/FRt4/detection_loss‚2.908859349348966 epoch‚6 0loss/FRt3/assignment_loss‚1.0806345076937425 /loss/FRt4/assignment_loss‚3.421234402586432 _step‚605 0loss/FRt2/assignment_loss‚2.7462449448053228 1loss/SRqqt1/detection_loss‚1.6614709897917144 2loss/SRqqt1/assignment_loss‚1.6372089483300034 2loss/FBt3/assignment_loss‚0.019396819393707545 /loss/FBt1/detection_loss‚2.8224324136972427 0loss/FBt3/detection_loss‚0.43072811061957555 %loss/total_loss‚41.43470311994132c~7b‚!Ü 2loss/FBt3/assignment_loss‚0.019396819393707545 /loss/FBt1/detection_loss‚2.8224324136972427 0loss/FBt3/detection_loss‚0.43072811061957555 %loss/total_loss‚41.43470311994132 2loss/SRqqt4/assignment_loss‚2.2984670859116774 0loss/SRqqt2/detection_loss‚2.592668411321938 1loss/SRqqt3/detection_loss‚0.7537170914425091 2loss/SRqqt2/assignment_loss‚1.6663930359645747 1loss/FBt1/assignment_loss‚0.14224880363872217 .loss/FRt2/detection_loss‚2.029809320369887 /loss/FBt4/detection_loss‚2.2274883730070933 % _timestamp‚1.7884725091300516e+09 _runtime‚8562.315005251 0loss/FBt2/assignment_loss‚0.3174175227184375 /loss/FRt3/detection_loss‚1.3364764419139217 /loss/SRqqt4/detection_loss‚2.49925775711353 .loss/FBt2/detection_loss‚1.940285180744372 trainer/global_step‚7349 2loss/SRqqt3/assignment_loss‚1.0795084808487445 1loss/FBt4/assignment_loss‚0.21581304851653219 0loss/FRt1/assignment_loss‚2.9703765398747213 /loss/FRt1/detection_loss‚2.6365655402879455 .loss/FRt4/detection_loss‚2.908859349348966 epoch‚6 0loss/FRt3/assignment_loss‚1.0806345076937425 /loss/FRt4/assignment_loss‚3.421234402586432 _step‚605 0loss/FRt2/assignment_loss‚2.7462449448053228 1loss/SRqqt1/detection_loss‚1.6614709897917144 2loss/SRqqt1/assignment_loss‚1.6372089483300034ÝŽ·†fœƒ!‚(:‘ ¿ÙçÔªüƈ gpu.0.gpu‚38.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚104.161' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚26.040249999999997 gpu.0.smClock‚1230 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cƒ±t„!‚(:é ¿ÙçÔ‚щ disk.rbd0.out‚ 2.2421875 disk.sda1.out‚0 disk.sda2.in‚ 4.40234375 disk.sda2.out‚ 939.23828125" proc.memory.rssMB‚ 3032.2265625) proc.memory.percent‚18.50724220275879 disk.sdb.in‚ 4.48828125 disk.sda.out‚ 939.23828125# disk.nvme0n1.out‚50366.98828125 disk.md0p1.out‚ 905.31640625 cpu‚25.201246881255607$ disk./.usageGB‚2974.594009399414# disk.nvme0n1.in‚329855.34765625 disk.md0.out‚ 905.31640625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.dm-0.out‚ 109279.1875 disk.sdb2.out‚ 939.23828125$ memory_percent‚37.18132972717285 disk.rbd2.in‚0# disk.nvme1n1.in‚201313.06640625 disk.md0.in‚8.890625) proc.memory.availableMB‚ 10292.2109375 disk.rbd2.out‚ 2.466796875 disk.sdb.out‚ 939.23828125 disk.sda.in‚ 4.40234375 disk.dm-0.in‚531168.4140625 network.sent‚ 1.9164524e+07 network.recv‚ 2.940432e+06 disk.sda1.in‚0 disk.md0p1.in‚8.890625 disk.rbd0.in‚0.015625# disk.nvme1n1.out‚58912.19921875 disk.sdb2.in‚ 4.48828125 proc.cpu.threads‚17( disk./.usagePercent‚83.1708934671192&°Jh‘…!‚(:† ÎÙçÔ褪‡ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚32.0 gpu.0.powerWatts‚88.061' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚22.01525 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ûñ~†!‚(:ó ÎÙçÔ¸ó”ˆ disk.md0.in‚8.890625 disk.sda.out‚940.80615234375# disk.nvme1n1.in‚201313.06640625 disk.sdb.in‚ 4.48828125 disk.dm-0.in‚531168.4140625 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.sda1.in‚0% disk./.usageGB‚2974.8430709838867 disk.md0p1.out‚ 906.828125 disk.rbd0.in‚0.015625 disk.sdb.out‚940.80615234375 cpu‚25.20095975478338 disk.sda.in‚ 4.40234375# disk.nvme0n1.in‚329855.34765625! disk.sda2.out‚940.80615234375* proc.memory.availableMB‚10292.04296875 disk.sdb2.in‚ 4.48828125 disk.rbd2.in‚0 disk.sda1.out‚0 disk.rbd2.out‚ 2.466796875 network.sent‚ 1.925972e+07) disk./.usagePercent‚83.17785733326143! disk.sdb2.out‚940.80615234375 disk.md0.out‚ 906.828125 disk.rbd0.out‚ 2.2421875# disk.nvme1n1.out‚58912.30859375 disk.sdb1.out‚0! disk.nvme0n1.out‚ 50542.703125! disk.dm-0.out‚109455.01171875 disk.sda2.in‚ 4.40234375$ memory_percent‚37.18235492706299" proc.memory.rssMB‚ 3032.2265625) proc.memory.percent‚18.50724220275879 network.recv‚ 2.94776e+06 disk.md0p1.in‚8.890625£´ë=‡!‚(:… ÝÙçÔ“ÔÞ‡ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚32.0 gpu.0.powerWatts‚102.658' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.6645 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Â?Ô`€ˆ!‚(:õ ÝÙçÔ¹ºÉˆ disk.sdb.in‚ 4.48828125# disk.nvme0n1.out‚50743.64453125 disk.rbd2.in‚0 network.sent‚ 1.9267585e+07# disk.nvme1n1.in‚201313.06640625 disk.rbd2.out‚ 2.466796875 disk.sdb.out‚942.27001953125 disk.sdb1.in‚0 disk.md0.out‚ 908.23828125 disk.dm-0.out‚ 109656.171875$ memory_percent‚37.18242645263672) proc.memory.percent‚18.50724220275879 disk.sdb2.in‚ 4.48828125 disk.sda.out‚942.27001953125 disk.sda.in‚ 4.40234375 disk.dm-0.in‚531168.4140625 network.recv‚ 2.95155e+06) disk./.usagePercent‚83.17785679995878 disk.md0p1.out‚ 908.23828125 disk.rbd0.out‚ 2.2421875# disk.nvme1n1.out‚58912.52734375! disk.sda2.out‚942.27001953125% disk./.usageGB‚2974.8430519104004 disk.md0p1.in‚8.890625 disk.md0.in‚8.890625 disk.sdb1.out‚0 disk.sda1.out‚0 cpu‚25.20066580008849! disk.sdb2.out‚942.27001953125 disk.rbd0.in‚0.015625# disk.nvme0n1.in‚329855.34765625 disk.sda1.in‚0 disk.sda2.in‚ 4.40234375" proc.memory.rssMB‚ 3032.2265625 proc.cpu.threads‚17' proc.memory.availableMB‚ 10292.03125ÍåÎ>‘‰!‚(:† ìÙçÔ€ù¾‡ gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚104.494' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚26.1235 gpu.0.smClock‚1215 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`œ:pŠ!‚(:ö ìÙçÔíôŸˆ! disk.sda2.out‚944.25439453125 network.sent‚ 1.9272856e+07 disk.sdb2.in‚ 4.48828125! disk.sdb2.out‚944.25439453125 disk.md0p1.in‚8.890625 cpu‚25.200377395563898 disk.sda1.out‚0 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.sdb.out‚944.25439453125 disk.md0p1.out‚ 910.171875) disk./.usagePercent‚83.16311332792465 disk.rbd0.out‚ 2.2421875$ memory_percent‚37.18240261077881# disk.nvme1n1.in‚201313.06640625# disk.nvme0n1.out‚50821.16015625 disk.dm-0.out‚ 109733.828125 proc.cpu.threads‚17# disk.nvme1n1.out‚58912.66796875 disk.sdb1.in‚0# disk.nvme0n1.in‚329855.34765625 disk.sda1.in‚0 disk.sda2.in‚ 4.40234375 disk.sda.in‚ 4.40234375 disk.dm-0.in‚531168.4140625 network.recv‚ 2.955484e+06 disk.md0.in‚8.890625 disk.md0.out‚ 910.171875 disk.rbd2.out‚ 2.466796875 disk.sdb1.out‚0% disk./.usageGB‚2974.3157539367676* proc.memory.availableMB‚10292.03515625) proc.memory.percent‚18.50724220275879 disk.sdb.in‚ 4.48828125 disk.sda.out‚944.25439453125" proc.memory.rssMB‚ 3032.2265625.”2‹!- _wandbruntime‚8622  _runtime‚8622„!@lÇŒ!Á trainer/global_step‚7399 % _timestamp‚1.7884725697238455e+09 _runtime‚8622.908923527 _step‚606 & lr-AdamW/pg1‚0.0003237160901468791 & lr-AdamW/pg2‚0.0003237160901468791p ŸoÌ!Æ & lr-AdamW/pg1‚0.0003237160901468791 & lr-AdamW/pg2‚0.0003237160901468791 trainer/global_step‚7399 % _timestamp‚1.7884725697238455e+09 _runtime‚8622.908923527 _step‚606Þž·?Ñ2Ž!- _wandbruntime‚8622  _runtime‚8622!Lìn&!èO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚302099232409600 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14714073269760 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 137900881 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚334917896112128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reÆ•ÙÈ"served_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚302099250412544 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14672585587358 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚316813305679360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53541567 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚316813557350912 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 141061885 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 141061885 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14714306938368 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 137901573 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚334918206477312 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚287399781440336 _step‚607 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 201383558 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚302072138046610 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14672356606274 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 342471789 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 141061887 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚302099232409600 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚316813557350912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53542257 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _runtime‚8622.913634279 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 201383558 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84359314 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14962961860096 trainer/global_step‚7399 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14714073269760 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚316813305679360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚302072384067054 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚334918206477312 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚287399781440336 _step‚607 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 201383558 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚302072138046610 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14672356606274 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 342471789 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 141061887 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚302099232409600 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚316813557350912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53542257 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _runtime‚8622.913634279 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 201383558 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84359314 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14962961860096 trainer/global_step‚7399 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14714073269760 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚316813305679360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚302072384067054 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14714073269760 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 137900881 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚334917896112128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚302099250412544 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14672585587358 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚316813305679360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53541567 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚316813557350912 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 141061885 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496ßæÿMŸ2‘!- _wandbruntime‚8624  _runtime‚8624¬”qõø&’!òM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚319996904647168 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 % _timestamp‚1.7884725713801677e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 141080162 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚302138536449536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 201436620 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137918667 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚319996928587264 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 342490410 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚316854807427072 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 342516782 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚302138536449536 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14716270977536 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 342490410 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 342516782 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚287437328837120 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚302111872913672 _step‚608 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 201436620 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14716037295616 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14716270977536 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚287437311797760 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14674544076552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚302138518446592 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚302138518446592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14964957319168 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137918667 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚319996928587264 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 342490410 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚316854807427072 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 342516782 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚302138536449536 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14716270977536 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 342490410 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 342516782 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚287437328837120 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚302111872913672 _step‚608 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 201436620 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14716037295616 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14716270977536 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚287437311797760 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14674544076552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚302138518446592 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚302138518446592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14964957319168 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736àzfæ@2”!- _wandbruntime‚8624  _runtime‚8624–1„X•!Ò 0loss/FRt1/assignment_loss‚2.9840769938060214 2loss/SRqqt3/assignment_loss‚0.8528224845592043 1loss/SRqqt4/assignment_loss‚1.899899327141397 1loss/FBt2/assignment_loss‚0.18369675752844408 0loss/SRqqt2/detection_loss‚2.117035020800198 epoch‚6 1loss/FBt1/assignment_loss‚0.18261863135759476 -loss/FRt2/detection_loss‚2.24283336447661 .loss/FRt3/detection_loss‚1.295083300538466 0loss/FRt2/assignment_loss‚3.2413811100025973 .loss/FRt4/detection_loss‚2.743820743635297 _step‚609 % _timestamp‚1.7884725713828185e+09 2loss/SRqqt2/assignment_loss‚1.5567835058797808 0loss/SRqqt1/detection_loss‚1.974866033184762 0loss/SRqqt4/detection_loss‚2.472996127079515 0loss/FBt3/detection_loss‚0.42638815507683675 /loss/FBt4/detection_loss‚2.6804808229207993 %loss/total_loss‚41.89602990124585 0loss/FRt3/assignment_loss‚1.0751668281958138 .loss/FRt1/detection_loss‚2.642893134696143 2loss/FBt3/assignment_loss‚0.050769855779060506 /loss/FBt1/detection_loss‚2.6092936056000844 trainer/global_step‚7399 _runtime‚ 8624.57153288 0loss/FRt4/assignment_loss‚3.1697809156030416 2loss/SRqqt1/assignment_loss‚1.8762607802363003 /loss/FBt4/assignment_loss‚0.568656111136078 1loss/SRqqt3/detection_loss‚0.9116765855267497 .loss/FBt2/detection_loss‚2.1367497064850550aA|]–!× 2loss/FBt3/assignment_loss‚0.050769855779060506 /loss/FBt1/detection_loss‚2.6092936056000844 trainer/global_step‚7399 _runtime‚ 8624.57153288 0loss/FRt4/assignment_loss‚3.1697809156030416 2loss/SRqqt1/assignment_loss‚1.8762607802363003 /loss/FBt4/assignment_loss‚0.568656111136078 1loss/SRqqt3/detection_loss‚0.9116765855267497 .loss/FBt2/detection_loss‚2.136749706485055 0loss/FRt1/assignment_loss‚2.9840769938060214 2loss/SRqqt3/assignment_loss‚0.8528224845592043 1loss/SRqqt4/assignment_loss‚1.899899327141397 1loss/FBt2/assignment_loss‚0.18369675752844408 0loss/SRqqt2/detection_loss‚2.117035020800198 epoch‚6 1loss/FBt1/assignment_loss‚0.18261863135759476 -loss/FRt2/detection_loss‚2.24283336447661 .loss/FRt3/detection_loss‚1.295083300538466 0loss/FRt2/assignment_loss‚3.2413811100025973 .loss/FRt4/detection_loss‚2.743820743635297 _step‚609 % _timestamp‚1.7884725713828185e+09 2loss/SRqqt2/assignment_loss‚1.5567835058797808 0loss/SRqqt1/detection_loss‚1.974866033184762 0loss/SRqqt4/detection_loss‚2.472996127079515 0loss/FBt3/detection_loss‚0.42638815507683675 /loss/FBt4/detection_loss‚2.6804808229207993 %loss/total_loss‚41.89602990124585 0loss/FRt3/assignment_loss‚1.0751668281958138 .loss/FRt1/detection_loss‚2.642893134696143áf“n—!‚(:… ûÙçÔ¦²Éˆ gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚103.544' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.886 gpu.0.smClock‚1200 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/ÛŸ˜!‚(:ö ûÙçԮ庉% disk./.usageGB‚2974.3154830932617 disk.md0.out‚ 911.4609375# disk.nvme1n1.out‚58912.94921875 disk.sdb.out‚945.59228515625# disk.nvme0n1.in‚329855.34765625 disk.dm-0.in‚531168.4140625" proc.memory.rssMB‚ 3032.2265625 disk.md0p1.in‚8.890625 disk.rbd2.in‚0 disk.sdb.in‚ 4.48828125* proc.memory.availableMB‚10292.01171875 cpu‚25.20005176422988 network.sent‚ 1.927591e+07 disk.md0p1.out‚ 911.4609375 disk.rbd0.out‚ 2.2421875 disk.dm-0.out‚ 109849.796875 network.recv‚ 2.959081e+06! disk.sda2.out‚945.59228515625! disk.sdb2.out‚945.59228515625 disk.sdb1.in‚0 disk.sdb1.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚37.18254566192627 disk.rbd2.out‚ 2.466796875 disk.sda.out‚945.59228515625) proc.memory.percent‚18.50724220275879 disk.sdb2.in‚ 4.48828125 disk.md0.in‚8.890625# disk.nvme1n1.in‚201313.06640625 disk.sda.in‚ 4.40234375# disk.nvme0n1.out‚50936.84765625) disk./.usagePercent‚83.16310575502686 disk.sda2.in‚ 4.40234375 disk.rbd0.in‚0.015625‹¬‹ïœ™!‚(:‘ ŠÚçÔô›Ú‡ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚104.161' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚26.040249999999997 gpu.0.smClock‚1215 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ß|š!‚(:ñ ŠÚçÔ¦½›ˆ# disk.nvme0n1.out‚51080.79296875 disk.sda.in‚ 4.40234375) proc.memory.availableMB‚ 10291.9765625" proc.memory.rssMB‚ 3032.2265625% disk./.usageGB‚2974.8423461914062 disk.dm-0.in‚531169.45703125 disk.md0.in‚8.890625 disk.rbd2.out‚ 2.466796875$ memory_percent‚37.18276023864746) proc.memory.percent‚18.50724220275879 disk.sda1.out‚0 disk.sda2.out‚947.0927734375 disk.md0p1.out‚ 912.9140625 disk.md0.out‚ 912.9140625 disk.nvme1n1.in‚ 201313.65625 disk.sdb.out‚947.0927734375 disk.sdb1.in‚0 disk.sdb1.out‚0# disk.nvme0n1.in‚329855.80078125! disk.dm-0.out‚109994.40234375 disk.rbd2.in‚0! disk.nvme1n1.out‚ 58913.609375 disk.sda.out‚947.0927734375 disk.sda1.in‚0 disk.sdb2.in‚ 4.48828125 disk.sdb2.out‚947.0927734375 disk.rbd0.out‚ 2.2421875 network.recv‚ 2.966432e+06) disk./.usagePercent‚83.17783706776034 disk.sda2.in‚ 4.40234375 disk.md0p1.in‚8.890625 cpu‚25.199742249385434 proc.cpu.threads‚17 network.sent‚ 1.9386161e+07 disk.rbd0.in‚0.015625 disk.sdb.in‚ 4.48828125-|Ÿ®w›!‚(:ì ™ÚçÔÞ詇 disk.rbd2.in‚0 disk.nvme1n1.in‚ 201313.65625 disk.sda.in‚ 4.40234375 disk.sda.out‚ 948.646484375# disk.nvme0n1.in‚329855.80078125 network.sent‚ 1.9391989e+07 disk.dm-0.in‚531169.45703125 disk.md0p1.in‚8.890625 disk.md0.out‚ 914.41015625 disk.md0.in‚8.890625" proc.memory.rssMB‚ 3032.2265625 proc.cpu.threads‚17 disk.sda1.in‚0 disk.md0p1.out‚ 914.41015625% disk./.usageGB‚2974.8737716674805 disk.rbd0.out‚ 2.2421875 disk.sdb.in‚ 4.48828125 cpu‚25.19946026309149 disk.sda2.out‚ 948.646484375 disk.sdb2.out‚ 948.646484375$ memory_percent‚37.18128204345703' proc.memory.availableMB‚ 10292.21875) proc.memory.percent‚18.50724220275879 disk.sdb.out‚ 948.646484375! disk.nvme1n1.out‚ 58913.796875# disk.nvme0n1.out‚51155.11328125! disk.dm-0.out‚110068.91015625 disk.sda2.in‚ 4.40234375) disk./.usagePercent‚83.17871573722394 disk.sdb2.in‚ 4.48828125 disk.sda1.out‚0 disk.sdb1.in‚0 disk.sdb1.out‚0 network.recv‚ 2.970706e+06 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 2.466796875ÆÖfÓœ!‚(:… ™ÚçÔ„žû‡ gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.228' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚27.057 gpu.0.smClock‚1230 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„¦’!‚(:‡ ¨ÚçÔÌ—¢ˆ gpu.0.gpu‚11.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚107.279' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚26.81975 gpu.0.smClock‚1215 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NœÓxž!‚(:í ¨ÚçÔ¤‡áˆ disk.rbd2.in‚0 disk.rbd2.out‚ 2.466796875 disk.sda1.out‚0 disk.md0p1.out‚916.3125% disk./.usageGB‚2975.0928955078125 disk.md0.in‚8.890625 disk.sda.in‚ 4.40234375* proc.memory.availableMB‚10292.22265625 disk.rbd0.in‚0.015625 disk.md0p1.in‚8.890625 disk.sdb2.in‚ 4.48828125" proc.memory.rssMB‚ 3032.2265625 disk.dm-0.in‚531169.45703125! disk.sda2.out‚950.60888671875 disk.sdb.in‚ 4.48828125 network.recv‚ 2.974924e+06 disk.rbd0.out‚ 2.2421875 disk.nvme1n1.in‚ 201313.65625 disk.sda.out‚950.60888671875" disk.nvme0n1.out‚ 51304.6328125 disk.sdb.out‚950.60888671875 disk.dm-0.out‚ 110218.53125! disk.sdb2.out‚950.60888671875" disk.nvme1n1.out‚ 58913.8984375 network.sent‚ 1.9397932e+07 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0$ memory_percent‚37.18125820159912# disk.nvme0n1.in‚329855.80078125 disk.sda2.in‚ 4.40234375) proc.memory.percent‚18.50724220275879 proc.cpu.threads‚17) disk./.usagePercent‚83.18484253151009 disk.md0.out‚916.3125 cpu‚25.199194000399277‘œA2Ÿ!- _wandbruntime‚8684  _runtime‚8684"Κ†É !à ' lr-AdamW/pg1‚0.00031791335041740594 ' lr-AdamW/pg2‚0.00031791335041740594 trainer/global_step‚7449 % _timestamp‚1.7884726313981586e+09 _runtime‚8684.583153493 _step‚6109Ž/Ρ!È ' lr-AdamW/pg1‚0.00031791335041740594 ' lr-AdamW/pg2‚0.00031791335041740594 trainer/global_step‚7449 % _timestamp‚1.7884726313981586e+09 _runtime‚8684.583153493 _step‚610âÕ ¼œ2¢!- _wandbruntime‚8684  _runtime‚8684A_ŽSî'£!èO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚7449 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14812288976384 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 % _timestamp‚1.7884726313990443e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 202721510 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚337102389032448 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 141975637 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚318875823235584 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚304063552262144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚697 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53888499 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15062659487744 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚304063552262144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚304063534259200 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84899414 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚318876074907136 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14770294646782 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚695 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14812522644992 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 138787218 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚8684.587833045 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 344723491 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 141975635 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 141975635 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 202721510 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14812288976384 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚304063534259200 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 202747854 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚318876074907136 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141975637 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚322039443119616 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 202747854 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 138787915 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15062945912832 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚322039419179520 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚289276299311536 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚337102078667264 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14770523627866 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14812522644992 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚322039419179520 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚289276299311536 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚337102078667264 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14770523627866 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14812522644992 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚318875823235584 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚304063552262144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚697 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53888499 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15062659487744 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJöR€DeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚304063552262144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚304063534259200 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84899414 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚318876074907136 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14770294646782 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚695 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14812522644992 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 138787218 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚8684.587833045 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 344723491 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 141975635 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0ãW:’G’¥!‚(:‡ ·ÚçÔ¦Ñʇ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚107.279' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚26.81975 gpu.0.smClock‚1200 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%çÓa}¦!‚(:ò ·ÚçÔ¿š‰ network.recv‚ 2.979026e+06$ disk./.usageGB‚2975.093437194824 disk.rbd0.in‚0.015625 cpu‚25.19887723989597 disk.nvme1n1.in‚ 201313.65625# disk.nvme1n1.out‚58914.12890625 disk.sdb.in‚ 4.48828125 disk.sdb1.in‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 3032.2265625 disk.sdb2.in‚ 4.48828125 disk.md0.in‚8.890625 disk.sda.in‚ 4.40234375 disk.sda.out‚951.95361328125 disk.sda2.in‚ 4.40234375 disk.md0p1.out‚ 917.609375 disk.md0.out‚ 917.609375 disk.dm-0.in‚531169.45703125 disk.dm-0.out‚ 110368.484375 disk.md0p1.in‚8.890625* proc.memory.availableMB‚10291.98046875 network.sent‚ 1.9413871e+07! disk.sdb2.out‚951.95361328125 disk.rbd0.out‚ 2.2421875# disk.nvme0n1.in‚329855.80078125# disk.nvme0n1.out‚51454.35546875) disk./.usagePercent‚83.18485767730564 disk.rbd2.in‚0 disk.sdb.out‚951.95361328125 disk.sda1.in‚0! disk.sda2.out‚951.95361328125$ memory_percent‚37.18273639678955 proc.cpu.threads‚17 disk.rbd2.out‚ 2.466796875 disk.sdb1.out‚0) proc.memory.percent‚18.50724220275879™˜wQ2§!- _wandbruntime‚8685  _runtime‚86855ba‹÷&¨!ñM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 141993912 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53895267 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15064940008448 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚304102838299136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚337145754649088 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 202774572 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138804802 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚304102838299136 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚318917073298432 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15064651499520 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚612 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141993910 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14814253002240 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14814486684160 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 202774572 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84910218 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 141993912 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 138805485 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14772482117060 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚322081127089664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚683 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14772253135768 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14814486684160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚681 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚322081103149568 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84910216 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚289313829668960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 202748202 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 344742112 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚304086082804728 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚7449 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 $ _timestamp‚1.788472632611141e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 344768484 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _runtime‚8685.797948543 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 202748202 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14814253002240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 344742112 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273pU0®ü&©!öM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚304102820296192 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚318917073298432 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15064651499520 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚612 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141993910 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14814253002240 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14814486684160 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 202774572 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84910218 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 141993912 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 138805485 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14772482117060 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚322081127089664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚683 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14772253135768 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14814486684160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚681 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚322081103149568 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84910216 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚289313829668960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 202748202 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 344742112 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚304086082804728 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚7449 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 $ _timestamp‚1.788472632611141e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 344768484 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _runtime‚8685.797948543 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 202748202 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14814253002240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 344742112 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 141993912 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53895267 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15064940008448 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚304102838299136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚337145754649088 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 202774572 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138804802 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚304102838299136 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚320938592463360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚306021049870026 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14910738351616 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14868232687290 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚306027836108800 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚339286395142656 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚291152834222096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54234735 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884726932969263e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142889385 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15162753520640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚85439516 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚320938340791808 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚306027836108800 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85439514 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚615 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚306027854111744 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚320938340791808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 trainer/global_step‚7499 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 139674249 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚320938592463360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142889387 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚306021295890470 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 204085806¨Gwó'º!íO G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚306027854111744 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚324123617681920 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 204059462 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 142889387 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54235442 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14910504683008 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14910738351616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚291152817182736 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14910504683008 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚615 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚306027854111744 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚320938340791808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 trainer/global_step‚7499 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 139674249 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚320938592463360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142889387 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚306021295890470 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 204085806 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14868461668374 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 _runtime‚8746.482330888 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 204085806 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 139674958 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚339286084777472 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15162467095552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚320938592463360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚306021049870026 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14910738351616 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14868232687290 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚306027836108800 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚339286395142656 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚291152834222096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54234735 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884726932969263e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142889385 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15162753520640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚85439516 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚320938340791808 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚306027836108800 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85439514 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 346993814 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 204086154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 347020186 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚339330093533696 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142907662 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85450318 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚320979842539520 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚306060538716436 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚306060784737088 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚306067122145792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54241509 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15164481529856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14870420157568 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚685 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚306067140148736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14912702390784 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚291190347540160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14912468708864 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 139692510 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14870191176276 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚8747.674215101 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 347020186 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚306067122145792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚320979842539520 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142907660 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15164767941632 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 204112524 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 346993814 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚320979590854656 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142907662 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚291190364579520 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139691825 % _timestamp‚1.7884726944882581e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14912702390784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 204086154Z 9ÓB½!÷M >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚H¿j¬»&582 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚291190364579520 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139691825 % _timestamp‚1.7884726944882581e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14912702390784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 204086154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚324165325592064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 346993814 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 204086154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 347020186 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚339330093533696 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142907662 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85450318 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚320979842539520 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚306060538716436 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚306060784737088 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚306067122145792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54241509 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15164481529856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14870420157568 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚685 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚306067140148736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14912702390784 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚291190347540160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14912468708864 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 139692510 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14870191176276 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚8747.674215101 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 347020186 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚306067122145792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚320979842539520 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142907660 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15164767941632 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 204112524 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 346993814 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚320979590854656 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142907662è]ãN2¾!- _wandbruntime‚8747  _runtime‚8747 wÄcY¿!Ó 2loss/SRqqt3/assignment_loss‚1.1050478519339646 $ _timestamp‚1.788472694491418e+09 epoch‚6 0loss/SRqqt4/assignment_loss‚1.96517880239657 2loss/FBt2/assignment_loss‚0.045594219446979875 .loss/FRt3/detection_loss‚1.585825981625489 1loss/SRqqt4/detection_loss‚2.1727952003479003 _step‚617 1loss/SRqqt3/detection_loss‚0.7348698949707406 %loss/total_loss‚40.99240045834479 .loss/FBt1/detection_loss‚2.467775331603156 /loss/FRt1/detection_loss‚2.6811542340687344 /loss/FRt2/assignment_loss‚3.447383403778076 2loss/SRqqt2/assignment_loss‚1.4805976014894744 2loss/FBt1/assignment_loss‚0.053968832462689735 0loss/SRqqt2/detection_loss‚2.341886087258657 .loss/FBt2/detection_loss‚2.173309329152107 0loss/FRt3/assignment_loss‚1.0735356756619046 -loss/FRt4/detection_loss‚3.16651850938797 0loss/SRqqt1/detection_loss‚2.068397236906964 0loss/FBt3/detection_loss‚0.43104010067259274 /loss/FBt4/detection_loss‚2.6603010296821594 .loss/FRt2/detection_loss‚2.195244866206722 /loss/FRt4/assignment_loss‚2.806888605867113 2loss/SRqqt1/assignment_loss‚1.6123582702939925 1loss/FBt3/assignment_loss‚0.08018890370429896 1loss/FBt4/assignment_loss‚0.38833436334969207 _runtime‚8747.676335971 0loss/FRt1/assignment_loss‚2.2542061260768347 trainer/global_step‚7499¿Ö·^À!Ø /loss/FRt1/detection_loss‚2.6811542340687344 /loss/FRt2/assignment_loss‚3.447383403778076 2loss/SRqqt2/assignment_loss‚1.4805976014894744 2loss/FBt1/assignment_loss‚0.053968832462689735 0loss/SRqqt2/detection_loss‚2.341886087258657 .loss/FBt2/detection_loss‚2.173309329152107 0loss/FRt3/assignment_loss‚1.0735356756619046 -loss/FRt4/detection_loss‚3.16651850938797 0loss/SRqqt1/detection_loss‚2.068397236906964 0loss/FBt3/detection_loss‚0.43104010067259274 /loss/FBt4/detection_loss‚2.6603010296821594 .loss/FRt2/detection_loss‚2.195244866206722 /loss/FRt4/assignment_loss‚2.806888605867113 2loss/SRqqt1/assignment_loss‚1.6123582702939925 1loss/FBt3/assignment_loss‚0.08018890370429896 1loss/FBt4/assignment_loss‚0.38833436334969207 _runtime‚8747.676335971 0loss/FRt1/assignment_loss‚2.2542061260768347 trainer/global_step‚7499 2loss/SRqqt3/assignment_loss‚1.1050478519339646 $ _timestamp‚1.788472694491418e+09 epoch‚6 0loss/SRqqt4/assignment_loss‚1.96517880239657 2loss/FBt2/assignment_loss‚0.045594219446979875 .loss/FRt3/detection_loss‚1.585825981625489 1loss/SRqqt4/detection_loss‚2.1727952003479003 _step‚617 1loss/SRqqt3/detection_loss‚0.7348698949707406 %loss/total_loss‚40.99240045834479 .loss/FBt1/detection_loss‚2.467775331603156é_ØLHÁ!‚(:… ‚ÛçÔ‚“Õ‡ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚90.846' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚22.7115 gpu.0.smClock‚1215 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åÅ‚Í~Â!‚(:ó ‚ÛçÔ똢‰) proc.memory.availableMB‚ 10120.5703125 disk.md0p1.out‚ 925.953125 network.sent‚ 1.9625379e+07" disk.nvme1n1.out‚ 58914.8515625 disk.md0p1.in‚8.890625" proc.memory.rssMB‚ 3032.2265625) proc.memory.percent‚18.50724220275879 cpu‚25.197493365049617 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 2.2421875 disk.sdb.in‚ 4.48828125# disk.nvme0n1.in‚329919.79296875 disk.sda1.out‚0 disk.md0.out‚ 925.953125 proc.cpu.threads‚17 disk.rbd2.out‚ 2.466796875 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.sda.out‚960.61962890625% disk./.usageGB‚2974.7711334228516 disk.sdb.out‚960.61962890625 disk.sdb1.in‚0 disk.sda.in‚ 4.40234375! disk.sda2.out‚960.61962890625$ memory_percent‚38.22894096374512 network.recv‚ 3.005356e+06# disk.nvme0n1.out‚52220.59765625 disk.dm-0.in‚531281.10546875! disk.sdb2.out‚960.61962890625 disk.md0.in‚8.890625! disk.dm-0.out‚111135.44921875 disk.sda1.in‚0 disk.sda2.in‚ 4.40234375) disk./.usagePercent‚83.17584592894669 disk.nvme1n1.in‚ 201361.3125 disk.sdb2.in‚ 4.48828125†¯½bŽÃ!‚(:ƒ ‘ÛçÔ¨À¬‡ gpu.0.gpu‚38.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚105.72' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚26.43 gpu.0.smClock‚1215 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÊäuÄ!‚(:ê ‘ÛçÔ×àŒˆ disk.sda1.in‚0 disk.sdb2.in‚ 4.48828125 disk.nvme1n1.in‚ 201361.3125) disk./.usagePercent‚83.17786703936986 disk.dm-0.in‚531287.1328125 disk.rbd0.out‚ 2.2421875 disk.sdb.in‚ 4.48828125 disk.sda2.in‚ 4.40234375 disk.md0.out‚ 927.41015625 disk.rbd2.out‚ 2.466796875 disk.sdb1.out‚0" disk.nvme0n1.in‚329925.8203125" disk.nvme0n1.out‚ 52365.2109375& proc.memory.availableMB‚ 10120.8125" proc.memory.rssMB‚ 3032.2265625 disk.sda1.out‚0" disk.nvme1n1.out‚ 58915.2734375 disk.sdb.out‚962.12744140625) proc.memory.percent‚18.50724220275879 network.recv‚ 3.012502e+06 disk.md0p1.in‚8.890625 disk.sda.in‚ 4.40234375 network.sent‚ 1.972e+07$ disk./.usageGB‚2974.843418121338 disk.dm-0.out‚ 111280.484375 disk.md0p1.out‚ 927.41015625! disk.sdb2.out‚962.12744140625 disk.md0.in‚8.890625 disk.rbd0.in‚0.015625 disk.sda.out‚962.12744140625 disk.sdb1.in‚0$ memory_percent‚38.22746276855469 cpu‚25.197220793984197 proc.cpu.threads‚17! disk.sda2.out‚962.12744140625 disk.rbd2.in‚0Z |ÞÅ!‚(:„  ÛçԬķ‡ gpu.0.gpu‚34.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚90.236' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚22.559 gpu.0.smClock‚1200 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0oâ5IzÆ!‚(:ï  ÛçÔè塈 proc.cpu.threads‚17 network.recv‚ 3.016824e+06) disk./.usagePercent‚83.17961616543612$ disk./.usageGB‚2974.905975341797 disk.sdb.in‚ 4.48828125 disk.sdb2.in‚ 4.48828125" disk.nvme0n1.in‚329925.8203125# disk.nvme0n1.out‚52465.25390625 disk.rbd0.out‚ 2.2421875# memory_percent‚38.2274866104126 disk.sda1.in‚0 disk.md0p1.in‚8.890625 disk.md0.out‚ 928.87890625* proc.memory.availableMB‚10120.80859375 disk.nvme1n1.in‚ 201361.3125! disk.nvme1n1.out‚ 58915.390625 disk.sdb.out‚963.6552734375 disk.sda.out‚963.6552734375) proc.memory.percent‚18.50724220275879 disk.sda.in‚ 4.40234375 disk.rbd0.in‚0.015625 cpu‚25.196912881327254 disk.rbd2.in‚0! disk.dm-0.out‚111380.64453125 disk.sda2.in‚ 4.40234375" proc.memory.rssMB‚ 3032.2265625 network.sent‚ 1.9726015e+07 disk.sdb1.out‚0 disk.sda1.out‚0 disk.sdb2.out‚963.6552734375 disk.rbd2.out‚ 2.466796875 disk.sdb1.in‚0 disk.dm-0.in‚531287.1328125 disk.sda2.out‚963.6552734375 disk.md0p1.out‚ 928.87890625 disk.md0.in‚8.890625!Ë#dœÇ!‚(:‘ ¯ÛçÔġ gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚102.326' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚25.581499999999995 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§ƒVyvÈ!‚(:ë ¯ÛçÔÏÙ¬ˆ disk.md0p1.in‚8.890625* proc.memory.availableMB‚10120.80859375 network.recv‚ 3.020592e+06 disk.dm-0.in‚531287.1328125 disk.rbd0.out‚ 2.2421875# disk.nvme0n1.out‚52615.01171875! disk.sda2.out‚965.02392578125 proc.cpu.threads‚17! disk.sdb2.out‚965.02392578125 disk.rbd2.out‚ 2.466796875 disk.sdb.in‚ 4.48828125 disk.sdb1.in‚0 network.sent‚ 1.9733484e+07( disk./.usagePercent‚83.1848707965511 disk.sdb2.in‚ 4.48828125 disk.md0.in‚8.890625 disk.rbd0.in‚0.015625# disk.nvme1n1.out‚58915.62109375 disk.sda.in‚ 4.40234375" disk.nvme0n1.in‚329925.8203125 disk.sda1.out‚0 disk.sda2.in‚ 4.40234375$ disk./.usageGB‚2975.093906402588 disk.sdb.out‚965.02392578125 disk.sdb1.out‚0 disk.sda.out‚965.02392578125 disk.dm-0.out‚111530.6328125 disk.sda1.in‚0 disk.rbd2.in‚0 disk.nvme1n1.in‚ 201361.3125 disk.md0p1.out‚930.1875" proc.memory.rssMB‚ 3032.2265625) proc.memory.percent‚18.50724220275879 disk.md0.out‚930.1875# memory_percent‚38.2274866104126 cpu‚25.196634753604286v'd2É!- _wandbruntime‚8808  _runtime‚8808ä“úµÉÊ!à ' lr-AdamW/pg1‚0.00030636804020864463 ' lr-AdamW/pg2‚0.00030636804020864463 trainer/global_step‚7549 % _timestamp‚1.7884727554479809e+09 _runtime‚8808.632954627 _step‚618%`x9ÎË!È ' lr-AdamW/pg1‚0.00030636804020864463 ' lr-AdamW/pg2‚0.00030636804020864463 trainer/global_step‚7549 % _timestamp‚1.7884727554479809e+09 _runtime‚8808.632954627 _step‚618ê›ïŒ»2Ì!- _wandbruntime‚8808  _runtime‚8808`TIî'Í!èO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚307992155961344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 % _timestamp‚1.7884727554490488e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 205397414 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚85979616 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚293029352093296 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚341470504087552 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚293029335053936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 143803135 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚307995505781734 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚323000858348032 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14966170727798 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15008954058240 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36Ìý¥aÉ Î!íO [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15008720389632 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54583207 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚686 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15262663963136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 trainer/global_step‚7549 _runtime‚8808.637616998 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚307992137958400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚307995751802178 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 205423758 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15008954058240 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚307995505781734 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚323000858348032 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14966170727798 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15008954058240 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚307992155961344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 % _timestamp‚1.7884727554490488e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 205397414 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚85979616 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚293029352093296 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚341470504087552 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚293029335053936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 143803135 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 349271888 trainer/global_step‚7549 $ _timestamp‚1.788472756648304e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 205450476 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15010918097408 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15010684415488 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 205450476 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚308035240648796 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 _runtime‚8809.835500057 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 349245516 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 205424106 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140579940 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚701 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚341513886199808 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15010918097408 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚308031441998336 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 140580643 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚323042108410880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚293066865411360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54589524 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚326249524094464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚308034994628144 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚620 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0¯ÁEü&Ñ!öM 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚308031423995392 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 143821412 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚326249500154368 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚293066882450720 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14968358198076 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14968129216784 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 349271888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 143821410 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚323042360095744 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 143821410 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85990416 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 349271888 trainer/global_step‚7549 $ _timestamp‚1.788472756648304e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 205450476 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15010918097408 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15010684415488 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 205450476 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚308035240648796 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 _runtime‚8809.835500057 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 349245516 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 205424106 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140579940 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚701 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚341513886199808 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15010918097408 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚308031441998336 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 140580643 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚323042108410880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚293066865411360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54589524 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚326249524094464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚308034994628144 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚620 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496ìÀ»†2Ò!- _wandbruntime‚8809  _runtime‚8809† ^]Ó!× _step‚621 /loss/FRt2/assignment_loss‚3.015117133205587 2loss/SRqqt2/assignment_loss‚1.5502310930751264 2loss/SRqqt3/assignment_loss‚1.0060678551866051 .loss/FRt1/detection_loss‚2.836002154275775 /loss/FRt4/detection_loss‚2.9979906493219834 /loss/FBt2/detection_loss‚2.7617013433149884 /loss/FBt3/detection_loss‚0.5844758100903187 0loss/FRt1/assignment_loss‚2.8293705210089684 0loss/SRqqt2/detection_loss‚2.213129786774516 /loss/FBt1/detection_loss‚2.4509442363466536 1loss/FBt4/assignment_loss‚0.14752261861326968 1loss/SRqqt3/detection_loss‚0.8589181991604468 0loss/FRt3/assignment_loss‚1.3462926956323478 2loss/SRqqt1/assignment_loss‚1.3780789030715823 2loss/SRqqt4/assignment_loss‚1.5683702117261977 epoch‚6 _runtime‚8809.840531629 0loss/FRt4/assignment_loss‚3.0771853553837745 1loss/SRqqt4/detection_loss‚2.5278547406196594 1loss/FBt1/assignment_loss‚0.10471242353278447 .loss/FRt2/detection_loss‚2.142640405710236 1loss/SRqqt1/detection_loss‚2.0562511935830114 .loss/FBt4/detection_loss‚2.560896054903666 %loss/total_loss‚41.83072726943604 trainer/global_step‚7549 % _timestamp‚1.7884727566521022e+09 1loss/FBt2/assignment_loss‚0.10384083483570196 1loss/FBt3/assignment_loss‚0.06967613825137511 .loss/FRt3/detection_loss‚1.643456911811462îýFÄbÔ!Ü 1loss/FBt3/assignment_loss‚0.06967613825137511 .loss/FRt3/detection_loss‚1.643456911811462 _step‚621 /loss/FRt2/assignment_loss‚3.015117133205587 2loss/SRqqt2/assignment_loss‚1.5502310930751264 2loss/SRqqt3/assignment_loss‚1.0060678551866051 .loss/FRt1/detection_loss‚2.836002154275775 /loss/FRt4/detection_loss‚2.9979906493219834 /loss/FBt2/detection_loss‚2.7617013433149884 /loss/FBt3/detection_loss‚0.5844758100903187 0loss/FRt1/assignment_loss‚2.8293705210089684 0loss/SRqqt2/detection_loss‚2.213129786774516 /loss/FBt1/detection_loss‚2.4509442363466536 1loss/FBt4/assignment_loss‚0.14752261861326968 1loss/SRqqt3/detection_loss‚0.8589181991604468 0loss/FRt3/assignment_loss‚1.3462926956323478 2loss/SRqqt1/assignment_loss‚1.3780789030715823 2loss/SRqqt4/assignment_loss‚1.5683702117261977 epoch‚6 _runtime‚8809.840531629 0loss/FRt4/assignment_loss‚3.0771853553837745 1loss/SRqqt4/detection_loss‚2.5278547406196594 1loss/FBt1/assignment_loss‚0.10471242353278447 .loss/FRt2/detection_loss‚2.142640405710236 1loss/SRqqt1/detection_loss‚2.0562511935830114 .loss/FBt4/detection_loss‚2.560896054903666 %loss/total_loss‚41.83072726943604 trainer/global_step‚7549 % _timestamp‚1.7884727566521022e+09 1loss/FBt2/assignment_loss‚0.10384083483570196íÕÁ%‘Õ!‚(:† ¾ÛçÔ꣄‡ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚116.306' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.0765 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Iñ¥FmÖ!‚(:â ¾Ûçԉ橉$ memory_percent‚38.23075294494629 cpu‚25.196352288993694 disk.rbd2.out‚ 2.466796875 disk.sdb.out‚ 967.0390625 disk.sda.in‚ 4.40234375 disk.dm-0.in‚531287.1328125 proc.cpu.threads‚17 disk.rbd0.in‚0.015625 disk.nvme1n1.in‚ 201361.3125 disk.sda2.in‚ 4.40234375 disk.md0p1.in‚8.890625 disk.md0p1.out‚ 932.1328125 disk.sdb2.out‚ 967.0390625) proc.memory.availableMB‚ 10120.2734375" proc.memory.rssMB‚ 3032.2265625 disk.sdb.in‚ 4.48828125 network.recv‚ 3.024903e+06# disk.nvme1n1.out‚58915.75390625 disk.sdb1.out‚0 disk.sda.out‚ 967.0390625# disk.nvme0n1.out‚52790.34765625 disk.rbd2.in‚0 disk.dm-0.out‚111706.1015625 disk.sdb2.in‚ 4.48828125 network.sent‚ 1.9750397e+07$ disk./.usageGB‚2975.344352722168 disk.md0.in‚8.890625 disk.rbd0.out‚ 2.2421875 disk.sdb1.in‚0" disk.nvme0n1.in‚329925.8203125) proc.memory.percent‚18.50724220275879) disk./.usagePercent‚83.19187338046649 disk.sda1.in‚0 disk.sda2.out‚ 967.0390625 disk.md0.out‚ 932.1328125 disk.sda1.out‚0u.Š£×!‚(:‚ ÍÛçÔ¢°ã‡ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚101.1' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.275 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÂÁ§èØ!‚(:í ÍÛçÔ„óψ network.recv‚ 3.032104e+06 disk.rbd2.in‚0 disk.sdb1.in‚0) proc.memory.availableMB‚ 10121.3046875 network.sent‚ 1.9847325e+07 disk.sda.out‚968.7080078125$ disk./.usageGB‚2975.343334197998 disk.sda1.in‚0 disk.sdb.in‚ 4.48828125 disk.sdb.out‚968.7080078125 disk.sda2.in‚ 4.40234375# disk.nvme0n1.out‚52939.96484375$ memory_percent‚38.22445869445801( disk./.usagePercent‚83.1918449021044 disk.md0p1.out‚ 933.73046875 disk.sdb2.out‚968.7080078125 disk.md0.in‚8.890625 disk.md0.out‚ 933.73046875 disk.rbd0.out‚ 2.2421875 disk.nvme1n1.in‚ 201361.3125 disk.sdb1.out‚0 disk.sda2.out‚968.7080078125# disk.nvme1n1.out‚58916.01953üº125" disk.nvme0n1.in‚329925.8203125" proc.memory.rssMB‚ 3032.2265625) proc.memory.percent‚18.50724220275879 disk.dm-0.in‚531287.1328125 disk.dm-0.out‚ 111855.984375 disk.sda1.out‚0 disk.sdb2.in‚ 4.48828125 disk.rbd2.out‚ 2.466796875 cpu‚25.19608069863895 proc.cpu.threads‚17 disk.md0p1.in‚8.890625 disk.rbd0.in‚0.015625 disk.sda.in‚ 4.40234375nÞ3®’Ù!‚(:‡ ÜÛçÔõ­Ô‡ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02„ÿk{Ú!‚(:ð ÜÛçÔØÀ“ˆ disk.rbd2.out‚ 2.466796875% disk./.usageGB‚2975.3431854248047# disk.nvme1n1.in‚201361.37109375 disk.sdb.out‚970.2705078125 network.recv‚ 3.035979e+06 disk.dm-0.in‚531287.19140625 disk.sda2.in‚ 4.40234375" proc.memory.rssMB‚ 3032.2265625 proc.cpu.threads‚17" disk.nvme0n1.in‚329925.8203125! disk.dm-0.out‚112057.26953125 disk.md0.in‚8.890625 disk.md0.out‚ 935.234375 disk.rbd2.in‚0 disk.sdb1.in‚0 disk.sda2.out‚970.2705078125# disk.nvme1n1.out‚58916.15234375 disk.sda.out‚970.2705078125) proc.memory.percent‚18.50724220275879 cpu‚25.195808699910295 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 2.2421875 disk.sda.in‚ 4.40234375$ memory_percent‚38.22460174560547 disk.sdb2.in‚ 4.48828125 disk.sdb.in‚ 4.48828125' proc.memory.availableMB‚ 10121.28125 disk.sdb1.out‚0 disk.md0p1.out‚ 935.234375 network.sent‚ 1.9853266e+07) disk./.usagePercent‚83.19184074234364" disk.nvme0n1.out‚ 53141.1171875 disk.md0p1.in‚8.890625 disk.sdb2.out‚970.2705078125IÄÛ!‚(:… ëÛçÔˆüˆ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚131.45' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.8625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•rcó„Ü!‚(:ù ëÛçÔ®°˜‰) disk./.usagePercent‚83.19184074234364 disk.sdb1.in‚0 disk.dm-0.in‚531287.19140625! disk.sdb2.out‚972.00732421875 disk.rbd0.out‚ 2.2421875 network.sent‚ 1.9858601e+07% disk./.usageGB‚2975.3431854248047 disk.sdb.in‚ 4.48828125 disk.sdb1.out‚0" disk.nvme0n1.in‚329925.8203125 disk.dm-0.out‚112259.0859375 disk.rbd0.in‚0.015625# disk.nvme0n1.out‚53342.81640625 disk.md0p1.in‚8.890625 disk.md0p1.out‚ 936.9140625 disk.md0.out‚ 936.9140625$ memory_percent‚38.22615146636963 cpu‚25.195416279532083 disk.rbd2.out‚ 2.466796875 disk.sda2.in‚ 4.40234375 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sda1.in‚0! disk.sda2.out‚972.00732421875 disk.sdb2.in‚ 4.48828125* proc.memory.availableMB‚10121.02734375) proc.memory.percent‚18.50724220275879# disk.nvme1n1.out‚58916.26953125 disk.sda.in‚ 4.40234375 disk.sda.out‚972.00732421875 disk.sda1.out‚0" proc.memory.rssMB‚ 3032.2265625 network.recv‚ 3.039852e+06# disk.nvme1n1.in‚201361.37109375 disk.md0.in‚8.890625 disk.sdb.out‚972.00732421875eµl2Ý!- _wandbruntime‚8869  _runtime‚8869¼ÌitÇÞ!Á & lr-AdamW/pg2‚0.0003006278757546995 trainer/global_step‚7599 % _timestamp‚1.7884728165853066e+09 _runtime‚8869.770383933 _step‚622 & lr-AdamW/pg1‚0.0003006278757546995AÕ7öÌß!Æ & lr-AdamW/pg1‚0.0003006278757546995 & lr-AdamW/pg2‚0.0003006278757546995 trainer/global_step‚7599 % _timestamp‚1.7884728165853066e+09 _runtime‚8869.770383933 _step‚622î!#‘2à!- _wandbruntime‚8869  _runtime‚8869ð#gÂë'á!åO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 351478597 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚7599 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 141450752 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15107169764864 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 206761710 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54931038 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚325063375904256 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 144716885 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 206735366 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15064108768306 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144716887 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚328292038626816 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚294905869964496 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15064337749390 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 141451460 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚706 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚325063375904256 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 144716885 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 206735366 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15064108768306 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144716887 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚328292038626816 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚294905869964496 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15064337749390 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 141451460 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚706 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 141450752 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104ï ¸ÉM2ã!- _wandbruntime‚8871  _runtime‚8871]=½ðä!ñM E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 351523590 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141468284 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚310009450539852 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 141468950 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚309995725844992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 351497218 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15364478767616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚310009696560504 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15108900122112 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144735160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚86530518 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15108900122112 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚294943400321920 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚325104877651968 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 206788428 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15364188161536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 $ _timestamp‚1.788472818085863e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144735162 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15109133804032 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚325104625967104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚86530516 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 206788428 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚325104877651968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 trainer/global_step‚7599 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54938432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 351523590 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 206762058 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚328333698656768 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15066067257292 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 206762058 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15066296238584 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 351497218 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚309995743847936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 144735162 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚328333722596864 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚343698201364480 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _runtime‚8871.271601628 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54937768 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚624 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚343697886818304‰ìü&å!öM $ _timestamp‚1.788472818085863e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144735162 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15109133804032 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚325104625967104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚86530516 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 206788428 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚325104877651968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 trainer/global_step‚7599 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54938432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 351523590 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 206762058 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚328333698656768 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15066067257292 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 206762058 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15066296238584 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 351497218 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚309995743847936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 144735162 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚328333722596864 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚343698201364480 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _runtime‚8871.271601628 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54937768 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚624 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚343697886818304 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 351523590 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141468284 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚310009450539852 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 141468950 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚309995725844992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 351497218 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15364478767616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚310009696560504 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15108900122112 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144735160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚86530518 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15108900122112 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚294943400321920 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚325104877651968 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 206788428 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15364188161536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0ð–!b’2æ!- _wandbruntime‚8871  _runtime‚8871ó¨B¾Yç!Ó 2loss/FBt3/assignment_loss‚0.019626060507119958 1loss/FBt2/assignment_loss‚0.22610383380819757 -loss/FRt2/detection_loss‚2.18453442070964 1loss/SRqqt1/detection_loss‚1.9452809128496382 % _timestamp‚1.7884728180890326e+09 2loss/SRqqt1/assignment_loss‚1.4979747289584742 1loss/SRqqt4/assignment_loss‚1.710729857392866 _runtime‚8871.275372625 2loss/SRqqt2/assignment_loss‚1.3952204409986735 1loss/SRqqt3/detection_loss‚0.7807817393035259 .loss/FBt2/detection_loss‚1.789605091015498 /loss/FBt3/detection_loss‚0.5919985585020516 1loss/FBt4/assignment_loss‚0.23705739634370307 /loss/FRt3/detection_loss‚1.1500568013536194 .loss/FBt1/detection_loss‚2.419762259721756 .loss/FBt4/detection_loss‚3.080962210893631 %loss/total_loss‚41.05572948894833 trainer/global_step‚7599 _step‚625 /loss/FRt2/assignment_loss‚3.388053304270694 /loss/FRt3/assignment_loss‚1.261436106407479 2loss/SRqqt3/assignment_loss‚0.8598281446633473 epoch‚6 0loss/FRt4/assignment_loss‚2.8558322531836375 1loss/FBt1/assignment_loss‚0.14213739748765875 /loss/FRt4/detection_loss‚3.4969043107259843 1loss/SRqqt2/detection_loss‚2.1876555293798448 0loss/SRqqt4/detection_loss‚2.280129309358268 .loss/FRt1/detection_loss‚2.878795782725016 /loss/FRt1/assignment_loss‚2.675263038388005w O†^è!Ø 1loss/SRqqt4/assignment_loss‚1.710729857392866 _runtime‚8871.275372625 2loss/SRqqt2/assignment_loss‚1.3952204409986735 1loss/SRqqt3/detection_loss‚0.7807817393035259 .loss/FBt2/detection_loss‚1.789605091015498 /loss/FBt3/detection_loss‚0.5919985585020516 1loss/FBt4/assignment_loss‚0.23705739634370307 /loss/FRt3/detection_loss‚1.1500568013536194 .loss/FBt1/detection_loss‚2.419762259721756 .loss/FBt4/detection_loss‚3.080962210893631 %loss/total_loss‚41.05572948894833 trainer/global_step‚7599 _step‚625 /loss/FRt2/assignment_loss‚3.388053304270694 /loss/FRt3/assignment_loss‚1.261436106407479 2loss/SRqqt3/assignment_loss‚0.8598281446633473 epoch‚6 0loss/FRt4/assignment_loss‚2.8558322531836375 1loss/FBt1/assignment_loss‚0.14213739748765875 /loss/FRt4/detection_loss‚3.4969043107259843 1loss/SRqqt2/detection_loss‚2.1876555293798448 0loss/SRqqt4/detection_loss‚2.280129309358268 .loss/FRt1/detection_loss‚2.878795782725016 /loss/FRt1/assignment_loss‚2.675263038388005 2loss/FBt3/assignment_loss‚0.019626060507119958 1loss/FBt2/assignment_loss‚0.22610383380819757 -loss/FRt2/detection_loss‚2.18453442070964 1loss/SRqqt1/detection_loss‚1.9452809128496382 % _timestamp‚1.7884728180890326e+09 2loss/SRqqt1/assignment_loss‚1.4979747289584742ñ…ô d‘é!‚(:† úÛçÔàŒ•ˆ gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚129.614' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.4035 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²P¤+ê!‚(:ô úÛçÔ¼žñˆ* proc.memory.availableMB‚10121.24609375) proc.memory.percent‚18.50724220275879 disk.md0p1.out‚ 938.2734375! disk.sdb2.out‚973.42724609375 disk.sdb.in‚ 4.48828125 disk.sdb1.in‚0 disk.md0p1.in‚8.890625 disk.rbd0.in‚0.015625" disk.nvme1n1.in‚201367.1484375 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.19508920600853 disk.dm-0.in‚531319.7265625 disk.rbd2.out‚ 2.466796875 disk.md0.in‚8.890625# disk.nvme0n1.out‚53483.80078125$ memory_percent‚38.22481632232666 disk.sda2.in‚ 4.40234375 proc.cpu.threads‚17% disk./.usageGB‚2974.8156509399414 disk.rbd2.in‚0" disk.nvme1n1.out‚ 58916.3828125" proc.memory.rssMB‚ 3032.2265625 network.sent‚ 1.9861801e+07) disk./.usagePercent‚83.17709065735653 disk.md0.out‚ 938.2734375 disk.rbd0.out‚ 2.2421875 disk.sdb.out‚973.42724609375 disk.sda.in‚ 4.40234375! disk.sda2.out‚973.42724609375 network.recv‚ 3.04326e+06 disk.sdb2.in‚ 4.48828125 disk.sdb1.out‚0 disk.sda.out‚973.42724609375! disk.nvme0n1.in‚ 329952.578125! disk.dm-0.out‚112400.18359375r"©’ë!‚(:‡ ‰ÜçÔ••¾‡ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚133.349' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.33725 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ãäýøxì!‚(:í ‰ÜçÔòš¨ˆ disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda2.out‚974.8974609375 disk.md0p1.in‚8.890625 disk.nvme0n1.in‚ 329959.125 proc.cpu.threads‚17 network.sent‚ 1.9969414e+07 disk.sdb.out‚974.8974609375" disk.nvme0n1.out‚ 53556.0546875 disk.md0p1.out‚ 939.6796875" disk.nvme1n1.in‚201379.3828125 disk.dm-0.in‚531338.5078125$ memory_percent‚38.22636604309082) proc.memory.availableMB‚ 10120.9921875 disk.rbd2.out‚ 2.466796875$ disk./.usageGB‚2974.940647125244 disk.rbd0.in‚0.015625) proc.memory.percent‚18.50724220275879 network.recv‚ 3.050534e+06 disk.md0.out‚ 939.6796875# disk.nvme1n1.out‚59033.23828125 disk.sda.in‚ 4.40234375! disk.dm-0.out‚112589.29296875) disk./.usagePercent‚83.18058560301257 disk.rbd0.out‚ 2.2421875 disk.sda.out‚974.8974609375 disk.sda1.in‚0 disk.sda2.in‚ 4.40234375" proc.memory.rssMB‚ 3032.2265625 disk.sdb2.in‚ 4.48828125 cpu‚25.194820834554292 disk.sda1.out‚0 disk.sdb2.out‚974.8974609375 disk.md0.in‚8.890625 disk.rbd2.in‚0 disk.sdb.in‚ 4.48828125î_ùçœí!‚(:‘ ˜ÜçÔ§Å܇ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ä,eî!‚(:ö ˜ÜçÔ’隈 disk.md0.out‚ 941.22265625# disk.nvme1n1.out‚59105.26171875 disk.md0p1.in‚8.890625 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.md0p1.out‚ 941.22265625 network.recv‚ 3.055087e+06 disk.sdb2.in‚ 4.48828125 disk.sdb2.out‚976.4970703125 disk.sda.in‚ 4.40234375# disk.nvme0n1.in‚329959.16015625" disk.nvme0n1.out‚ 53628.6015625 disk.sda2.in‚ 4.40234375$ memory_percent‚38.22486400604248 network.sent‚ 1.9977745e+07" disk.nvme1n1.in‚201379.3828125 disk.sda.out‚976.4970703125! disk.dm-0.out‚112733.86328125* proc.memory.availableMB‚10121.23828125 disk.md0.in‚8.890625 disk.rbd0.out‚ 2.2421875 disk.rbd2.out‚ 2.466796875 disk.sdb.out‚976.4970703125 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚976.4970703125 disk.sdb.in‚ 4.48828125" proc.memory.rssMB‚ 3032.2265625 disk.rbd0.in‚0.015625 disk.sdb1.in‚0) proc.memory.percent‚18.50724220275879) disk./.usagePercent‚83.17780218976633 disk.dm-0.in‚531338.54296875 cpu‚25.19452265063132% disk./.usageGB‚2974.8410987854004TFIï!‚(:… §ÜçÔ€¤ˆ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚135.184' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.796 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0” Ëtoð!‚(:ä §ÜçÔ§‡Àˆ disk.sda2.out‚978.4716796875 disk.sdb.out‚978.4716796875# disk.nvme0n1.in‚329959.16015625) proc.memory.percent‚18.50724220275879 disk.md0.in‚10.96875" disk.nvme1n1.in‚201379.3828125 disk.sda.out‚978.4716796875 disk.dm-0.in‚531338.54296875 disk.sda1.in‚0$ memory_percent‚38.22488784790039( proc.memory.availableMB‚ 10121.234375 cpu‚25.194251070159133 disk.dm-0.out‚ 112806.15625 disk.md0.out‚ 943.13671875 disk.rbd2.in‚0 disk.sdb.in‚5.40625 disk.sdb1.in‚0 disk.sda.in‚5.5625$ disk./.usageGB‚2974.874568939209 disk.md0p1.in‚10.96875 disk.sdb2.out‚978.4716796875 network.recv‚ 3.058609e+06 disk.rbd0.in‚0.015625# disk.nvme1n1.out‚59105.39453125 disk.sda1.out‚0 disk.sda2.in‚5.5625 disk.md0p1.out‚ 943.13671875 disk.sdb2.in‚5.40625 disk.rbd2.out‚ 2.466796875 disk.sdb1.out‚0# disk.nvme0n1.out‚53700.76171875 proc.cpu.threads‚17 network.sent‚ 1.9983006e+07" proc.memory.rssMB‚ 3032.2265625) disk./.usagePercent‚83.17873802927515 disk.rbd0.out‚ 2.2421875(ïhF2ñ!- _wandbruntime‚8932  _runtime‚8932ÓäMtÆò!À & lr-AdamW/pg2‚0.0002949109682557425 trainer/global_step‚7649 % _timestamp‚1.7884728791501539e+09 _runtime‚ 8932.33521569 _step‚626 & lr-AdamW/pg1‚0.0002949109682557425Üû'Ëó!Å & lr-AdamW/pg1‚0.0002949109682557425 & lr-AdamW/pg2‚0.0002949109682557425 trainer/global_step‚7649 % _timestamp‚1.7884728791501539e+09 _runtime‚ 8932.33521569 _step‚626ò´vÙ2ô!- _wandbruntime‚8932  _runtime‚8932FZšíí'õ!çO G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚327125893460480 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15462294764032 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚327125893460480 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145630635 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚311920741657600 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 353730299 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 208073318 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87059816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚345838531893248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15205385471488 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚713 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 trainer/global_step‚7649 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚327126145132032 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15205151802880 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚345838208945152 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚8932.337095158 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 322948096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608ep5wö!ìO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚296782370796336 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚327125893460480 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145630635 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚311920741657600 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 353730299 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 208073318 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87059816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚345838531893248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15205385471488 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚713 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 trainer/global_step‚7649 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚327126145132032 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15205151802880 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚345838208945152 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚8932.337095158 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 322948096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚327125893460480 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15462294764032 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 $ _timestamp‚1.788472880345072e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15207115828736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚327167143523328 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15207349510656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚311960027694592 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145648912 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 142351893 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15164005297800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 208126380 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 353748920 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 208126380 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 142351225 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚296819901153760 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚327167395208192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚345881887339008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚330417921099264 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚311960045697536 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145648910 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 _runtime‚8933.531417589 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 208100010 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15207349510656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚666 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚296819918193120 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 208100010 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚330417897159168 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15164234279092 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15207115828736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 353775292 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15463990179840 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚628 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145648912 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚327167395208192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55281275 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚7649 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚668 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚345882203982336 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87070618 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87070616 –-ü&ù!öM ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15207349510656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚311960027694592 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145648912 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 142351893 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15164005297800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 208126380 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 353748920 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 208126380 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 142351225 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚296819901153760 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚327167395208192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚345881887339008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚330417921099264 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚311960045697536 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145648910 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 _runtime‚8933.531417589 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 208100010 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15207349510656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚666 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚296819918193120 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 208100010 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚330417897159168 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15164234279092 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15207115828736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 353775292 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15463990179840 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚628 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145648912 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚327167395208192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55281275 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚7649 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚668 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚345882203982336 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87070618 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87070616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 $ _timestamp‚1.788472880345072e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15207115828736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚327167143523328 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496ô’þv¿2ú!- _wandbruntime‚8933  _runtime‚8933Šà·äYû!Ó /loss/FRt2/assignment_loss‚3.602217197418213 1loss/SRqqt3/detection_loss‚0.8934218690512662 0loss/SRqqt4/detection_loss‚2.890763324719888 $ _timestamp‚1.788472880347997e+09 0loss/FRt3/assignment_loss‚0.9382393618424734 1loss/FBt1/assignment_loss‚0.17083132409969437 /loss/FRt3/detection_loss‚1.5966584811607996 %loss/total_loss‚40.58565497639632 _step‚629 1loss/FBt3/assignment_loss‚0.04321512330075713 trainer/global_step‚7649 2loss/SRqqt2/assignment_loss‚1.4170134986440341 /loss/FBt4/detection_loss‚2.5897279300472955 epoch‚6 _runtime‚8933.532758175 1loss/SRqqt4/assignment_loss‚1.309873603284359 1loss/FBt2/assignment_loss‚0.13991872484408166 .loss/FBt2/detection_loss‚1.785698073250907 0loss/FRt4/assignment_loss‚2.9010231097539267 /loss/FBt4/assignment_loss‚0.279162054521397 /loss/FRt4/detection_loss‚2.8782061517238615 .loss/FBt1/detection_loss‚2.279445670545101 /loss/FBt3/detection_loss‚0.7827581303470112 0loss/FRt1/assignment_loss‚2.7869836626908717 2loss/SRqqt1/assignment_loss‚1.5738283577090015 2loss/SRqqt3/assignment_loss‚0.7891745962872246 /loss/FRt1/detection_loss‚2.5377241709293465 0loss/SRqqt2/detection_loss‚2.564855385157797 .loss/FRt2/detection_loss‚2.085462883364276 1loss/SRqqt1/detection_loss‚1.7494522917027375ŸaJ^ü!Ø /loss/FRt2/assignment_loss‚3.602217197418213 1loss/SRqqt3/detection_loss‚0.8934218690512662 0loss/SRqqt4/detection_loss‚2.890763324719888 $ _timestamp‚1.788472880347997e+09 0loss/FRt3/assignment_loss‚0.9382393618424734 1loss/FBt1/assignment_loss‚0.17083132409969437 /loss/FRt3/detection_loss‚1.5966584811607996 %loss/total_loss‚40.58565497639632 _step‚629 1loss/FBt3/assignment_loss‚0.04321512330075713 trainer/global_step‚7649 2loss/SRqqt2/assignment_loss‚1.4170134986440341 /loss/FBt4/detection_loss‚2.5897279300472955 epoch‚6 _runtime‚8933.532758175 1loss/SRqqt4/assignment_loss‚1.309873603284359 1loss/FBt2/assignment_loss‚0.13991872484408166 .loss/FBt2/detection_loss‚1.785698073250907 0loss/FRt4/assignment_loss‚2.9010231097539267 /loss/FBt4/assignment_loss‚0.279162054521397 /loss/FRt4/detection_loss‚2.8782061517238615 .loss/FBt1/detection_loss‚2.279445670545101 /loss/FBt3/detection_loss‚0.7827581303470112 0loss/FRt1/assignment_loss‚2.7869836626908717 2loss/SRqqt1/assignment_loss‚1.5738283577090015 2loss/SRqqt3/assignment_loss‚0.7891745962872246 /loss/FRt1/detection_loss‚2.5377241709293465 0loss/SRqqt2/detection_loss‚2.564855385157797 .loss/FRt2/detection_loss‚2.085462883364276 1loss/SRqqt1/detection_loss‚1.7494522917027375õ.Æ’ý!‚(:‡ ¶ÜçÔÁ­Û‡ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚105.443' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚26.36075 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ú‰i ¦þ!‚(:ë ¶ÜçÔŒŽˆ disk.dm-0.out‚112983.7578125 disk.sda1.in‚0! disk.sda2.out‚980.02294921875 disk.sdb.out‚980.02294921875 disk.rbd2.in‚0" disk.nvme1n1.in‚201379.3828125" proc.memory.rssMB‚ 3032.2265625 disk.rbd0.out‚ 2.2421875# disk.nvme1n1.out‚59210.94921875) proc.memory.availableMB‚ 10121.1953125 disk.sdb.in‚5.40625 disk.sdb1.in‚0 disk.sdb1.out‚0# dis ŸçÐk.nvme0n1.in‚329959.16015625 disk.md0p1.out‚ 944.62109375! disk.sdb2.out‚980.02294921875 disk.rbd2.out‚ 2.466796875 cpu‚25.193984306838907 network.sent‚ 1.9987248e+07% disk./.usageGB‚2975.0937995910645 disk.sda.in‚5.5625# disk.nvme0n1.out‚53772.80859375 disk.dm-0.in‚531338.54296875 disk.sda1.out‚0 disk.sda2.in‚5.5625 disk.md0p1.in‚10.96875 disk.sda.out‚980.02294921875 disk.sdb2.in‚5.40625 disk.md0.in‚10.96875 disk.md0.out‚ 944.62109375 disk.rbd0.in‚0.015625$ memory_percent‚38.22512626647949) proc.memory.percent‚18.50724220275879 proc.cpu.threads‚17 network.recv‚ 3.06337e+06) disk./.usagePercent‚83.18486781005619ãTÛ’ÿ!‚(:… ÅÜçÔÊû¸‡ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚139.528' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«yJUv€"‚(:ë ÅÜçÔÆ‚¡ˆ network.recv‚ 3.070316e+06 disk.sda.in‚5.5625 disk.sda.out‚981.50634765625! disk.sda2.out‚981.50634765625 disk.rbd0.out‚ 2.2421875 disk.sdb.in‚5.40625" disk.nvme1n1.in‚201379.3828125# disk.nvme1n1.out‚59340.40234375 disk.sdb1.out‚0# disk.nvme0n1.in‚329959.16015625 disk.rbd2.out‚ 2.466796875 disk.sdb.out‚981.50634765625 disk.sdb1.in‚0 disk.dm-0.in‚531338.54296875 disk.sda1.in‚0$ memory_percent‚38.22529315948486 proc.cpu.threads‚17! disk.sdb2.out‚981.50634765625 disk.rbd0.in‚0.015625! disk.dm-0.out‚113185.42578125" proc.memory.rssMB‚ 3032.2265625) proc.memory.percent‚18.50724220275879 network.sent‚ 2.0096854e+07 disk.sdb2.in‚5.40625 disk.rbd2.in‚0 disk.sda1.out‚0 disk.md0p1.in‚10.96875* proc.memory.availableMB‚10121.16796875) disk./.usagePercent‚83.18485181097638" disk.nvme0n1.out‚ 53845.0234375 disk.md0p1.out‚ 946.0390625 cpu‚25.193687533377258 disk.sda2.in‚5.5625% disk./.usageGB‚2975.0932273864746 disk.md0.in‚10.96875 disk.md0.out‚ 946.0390625?˜D£‘"‚(:† ÔÜçÔýª“‡ gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚101.986' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.4965 gpu.0.smClock‚1170 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àÁ¥¯m‚"‚(:â ÔÜçÔ¨çŇ network.recv‚ 3.074677e+06( disk./.usagePercent‚83.1918419156095 disk.md0.in‚10.96875 disk.md0.out‚ 948.11328125* proc.memory.availableMB‚10121.16796875 cpu‚25.19339679024134 proc.cpu.threads‚17 disk.sda2.in‚5.5625 disk.rbd0.out‚ 2.2421875 disk.rbd2.out‚ 2.466796875) proc.memory.percent‚18.50724220275879 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.sda.in‚5.5625 disk.md0p1.in‚10.96875 network.sent‚ 2.0100695e+07# disk.nvme1n1.out‚59392.16015625 disk.sdb.in‚5.40625 disk.sda1.in‚0 disk.sda2.out‚ 983.650390625 disk.rbd0.in‚0.015625" disk.nvme1n1.in‚201379.3828125 disk.sdb.out‚ 983.650390625 disk.sdb1.in‚0! disk.nvme0n1.out‚ 53917.078125 disk.sda1.out‚0$ memory_percent‚38.22529315948486" proc.memory.rssMB‚ 3032.2265625 disk.sdb2.in‚5.40625 disk.sdb2.out‚ 983.650390625# disk.nvme0n1.in‚329959.16015625 disk.dm-0.in‚531338.54296875! disk.dm-0.out‚113309.23828125% disk./.usageGB‚2975.3432273864746 disk.sda.out‚ 983.650390625 disk.md0p1.out‚ 948.11328125e:Õ ƒ"‚(:„ ãÜçÔ’®‡ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚142.93' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.7325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿý…DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚329188411016704 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15303367509504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚313918873516858 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚313885043507200 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 209411270 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 209411270 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 209437614 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55621919 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚332460435631616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15562174885888 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚329188411016704 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚313885043507200 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚8995.827579781 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 trainer/global_step‚7699 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55621218 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15259984849322 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 % _timestamp‚1.7884729426422172e+09 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 355955655 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚703 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15260213830406 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 143221132 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15303601178112 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 146544387 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 146544385 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚313885061510144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 355982001 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87599916 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚348022298055168 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚313919119537302 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚298658888667536 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚313885061510144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚332460411691520 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15561886363648 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚329188662688256 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 146544385 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚348022610517504 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚298658905706896 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚329188662688256 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15303601178112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87599914 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582ÚâËó'Š"íO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 146544385 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚348022610517504 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚298658905706896 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚329188662688256 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15303601178112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87599914 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 143221835 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚329188411016704 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15303367509504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚313918873516858 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚313885043507200 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 209411270 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 209411270 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 209437614 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55621919 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚332460435631616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15562174885888 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚329188411016704 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚313885043507200 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚8995.827579781 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 trainer/global_step‚7699 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55621218 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15259984849322 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 % _timestamp‚1.7884729426422172e+09 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 355955655 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚703 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15260213830406 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 143221132 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15303601178112 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 146544387 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 146544385 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚313885061510144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 355982001 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87599916 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚348022298055168 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚313919119537302 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚298658888667536 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚313885061510144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚332460411691520 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15561886363648 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚329188662688256 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006÷Õ2xy2‹"- _wandbruntime‚8997  _runtime‚899725Ñ·Œ"òM REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55627983 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚348065977796096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15563882134528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚313924347547136 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚313924329544192 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚632 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 356026994 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚348066290245120 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87610716 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 356026994 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚693 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚695 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 209437962 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚298696436064320 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 356000622 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚332502095661568 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143238699 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15261943338308 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚7699 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 _runtime‚8997.036363415 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 146562660 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15305331535360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚313924347547136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚298696419024960 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 209464332 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 % _timestamp‚1.7884729438503056e+09 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚329229661079552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 143239394 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 356000622 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15564170643456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚313958608383920 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚329229912764416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚313958362363268 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 146562660 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15305565217280 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 209437962 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 146562662 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15305331535360c¸W¦ý&"÷M M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15261943338308 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚7699 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 _runtime‚8997.036363415 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 146562660 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15305331535360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚313924347547136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚298696419024960 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 209464332 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 % _timestamp‚1.7884729438503056e+09 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚329229661079552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 143239394 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 356000622 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15564170643456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚313958608383920 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚329229912764416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚313958362363268 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 146562660 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15305565217280 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 209437962 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 146562662 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15305331535360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55627983 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚348065977796096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15563882134528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚313924347547136 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚313924329544192 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚632 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 356026994 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚348066290245120 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87610716 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 356026994 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚693 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚695 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 209437962 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚298696436064320 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 356000622 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚332502095661568 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143238699 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0øI«Ó¦2Ž"- _wandbruntime‚8997  _runtime‚8997 ` L\"Ö .loss/FRt2/detection_loss‚2.633253031805907 /loss/FRt4/detection_loss‚3.3680961186235603 &loss/total_loss‚42.667176698177315 _step‚633 /loss/FRt4/assignment_loss‚2.880127164450559 2loss/SRqqt3/assignment_loss‚0.7625713705923839 1loss/SRqqt4/assignment_loss‚1.393470378825441 /loss/FBt1/detection_loss‚2.4036989395435038 /loss/FBt2/detection_loss‚2.7496906302191992 0loss/FRt1/assignment_loss‚2.6480074454756344 0loss/FBt3/assignment_loss‚0.0158304401936471 0loss/FRt3/assignment_loss‚1.3077435547655278 2loss/SRqqt2/assignment_loss‚1.8666159058610599 1loss/FBt1/assignment_loss‚0.28004994991336984 1loss/SRqqt1/detection_loss‚1.8997399667087866 /loss/FBt4/detection_loss‚2.1633860092414054 /loss/FRt3/detection_loss‚1.8074845679781653 0loss/SRqqt4/detection_loss‚2.229756511747837 % _timestamp‚1.7884729438539941e+09 _runtime‚8997.038767976 0loss/FBt2/assignment_loss‚0.2165546532969529 trainer/global_step‚7699 1loss/FBt4/assignment_loss‚0.16031663011474812 1loss/SRqqt2/detection_loss‚2.1336004048585893 1loss/SRqqt3/detection_loss‚0.8775084486438168 epoch‚6 /loss/FRt2/assignment_loss‚3.651162452995777 2loss/SRqqt1/assignment_loss‚1.6422319717097038 /loss/FBt3/detection_loss‚0.8002530453013605 .loss/FRt1/detection_loss‚2.776027105310384³š*a"Û /loss/FBt2/detection_loss‚2.7496906302191992 0loss/FRt1/assignment_loss‚2.6480074454756344 0loss/FBt3/assignment_loss‚0.0158304401936471 0loss/FRt3/assignment_loss‚1.3077435547655278 2loss/SRqqt2/assignment_loss‚1.8666159058610599 1loss/FBt1/assignment_loss‚0.28004994991336984 1loss/SRqqt1/detection_loss‚1.8997399667087866 /loss/FBt4/detection_loss‚2.1633860092414054 /loss/FRt3/detection_loss‚1.8074845679781653 0loss/SRqqt4/detection_loss‚2.229756511747837 % _timestamp‚1.7884729438539941e+09 _runtime‚8997.038767976 0loss/FBt2/assignment_loss‚0.2165546532969529 trainer/global_step‚7699 1loss/FBt4/assignment_loss‚0.16031663011474812 1loss/SRqqt2/detection_loss‚2.1336004048585893 1loss/SRqqt3/detection_loss‚0.8775084486438168 epoch‚6 /loss/FRt2/assignment_loss‚3.651162452995777 2loss/SRqqt1/assignment_loss‚1.6422319717097038 /loss/FBt3/detection_loss‚0.8002530453013605 .loss/FRt1/detection_loss‚2.776027105310384 .loss/FRt2/detection_loss‚2.633253031805907 /loss/FRt4/detection_loss‚3.3680961186235603 &loss/total_loss‚42.667176698177315 _step‚633 /loss/FRt4/assignment_loss‚2.880127164450559 2loss/SRqqt3/assignment_loss‚0.7625713705923839 1loss/SRqqt4/assignment_loss‚1.393470378825441 /loss/FBt1/detection_loss‚2.4036989395435038ù"yÝŽ‘"‚(:ƒ òÜçÔ¬˜•Œ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚114.74' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.685 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯–ñ}c’"‚(:Ø òÜçÔŠéΔ disk.md0p1.in‚10.96875 disk.md0.out‚ 951.2890625 disk.rbd0.out‚ 2.2421875 disk.rbd2.in‚0 cpu‚25.19270641364147 network.sent‚ 2.0112236e+07 disk.dm-0.in‚531338.54296875 disk.sda1.in‚0 disk.sdb2.out‚ 986.9609375* proc.memory.availableMB‚10121.07421875% disk./.usageGB‚2975.3432540893555 disk.sdb1.out‚0 disk.sda.out‚ 986.9609375 disk.md0.in‚10.96875) proc.memory.percent‚18.50724220275879 disk.sdb.out‚ 986.9609375 disk.sdb1.in‚0# disk.nvme0n1.out‚54061.23046875 disk.sda2.out‚ 986.9609375 disk.rbd2.out‚ 2.466796875 disk.sdb.in‚5.40625 disk.rbd0.in‚0.015625$ memory_percent‚38.22586536407471" proc.memory.rssMB‚ 3032.2265625) disk./.usagePercent‚83.19184266223323 disk.sda.in‚5.5625 disk.sda2.in‚5.5625! disk.nvme1n1.out‚ 59624.796875 network.recv‚ 3.08278e+06# disk.nvme0n1.in‚329959.16015625 disk.sda1.out‚0 disk.sdb2.in‚5.40625" disk.nvme1n1.in‚201379.3828125 proc.cpu.threads‚17! disk.dm-0.out‚113686.02734375 disk.md0p1.out‚ 951.2890625AÖŠÆ“"‚(:„ ÝçÔãœïŠ gpu.0.gpu‚8.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚121.876' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.469 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ægãhp”"‚(:å ÝçÔ¾£ó network.sent‚ 2.022188e+07) disk./.usagePercent‚83.19184266223323 disk.dm-0.in‚531338.54296875 disk.md0p1.in‚10.96875 disk.rbd0.out‚ 2.2421875 disk.sdb.out‚988.90185546875 cpu‚25.19240858227515 proc.cpu.threads‚17" disk.nvme0n1.out‚ 54133.3046875% memory_percent‚38.225483894348145 network.recv‚ 3.089807e+06 disk.sdb1.out‚0 disk.sda.out‚988.90185546875 disk.sda1.in‚0! disk.sdb2.out‚988.90185546875 disk.md0.out‚ 953.1640625 disk.nvme1n1.out‚ 59754.1875 disk.sda.in‚5.5625 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 2.466796875) proc.memory.percent‚18.50724220275879 disk.sda1.out‚0! disk.sda2.out‚988.90185546875 disk.sdb2.in‚5.40625 disk.rbd2.in‚0" disk.nvme1n1.in‚201379.3828125" proc.memory.rssMB‚ 3032.2265625% disk./.usageGB‚2975.3432540893555 disk.sdb.in‚5.40625* proc.memory.availableMB‚10121.13671875 disk.sdb1.in‚0# disk.nvme0n1.in‚329959.16015625 disk.dm-0.out‚113887.4921875 disk.sda2.in‚5.5625 disk.md0p1.out‚ 953.1640625 disk.md0.in‚10.96875³âñ’•"‚(:‡ ÝçÔÌèÇ gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚124.661' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.16525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…&fkv–"‚(:ë ÝçÔ¦Þ”ˆ" disk.nvme1n1.in‚201379.3828125" disk.nvme1n1.out‚ 59857.5703125 disk.sdb1.out‚0 disk.md0p1.in‚10.96875 disk.md0p1.out‚ 954.65234375* proc.memory.availableMB‚10120.88671875( disk./.usagePercent‚83.1918199435399 disk.md0.in‚10.96875 disk.sdb.out‚990.44189453125# disk.nvme0n1.out‚54205.35546875 disk.sda2.in‚5.5625! disk.sda2.out‚990.44189453125% memory_percent‚38.227009773254395 cpu‚25.19205774564009 disk.sda.in‚5.5625 disk.sda.out‚990.44189453125 network.recv‚ 3.094147e+06$ disk./.usageGB‚2975.342441558838! disk.sdb2.out‚990.44189453125 disk.md0.out‚ 954.65234375 disk.sdb.in‚5.40625 proc.cpu.threads‚17 disk.rbd0.in‚0.015625# disk.nvme0n1.in‚329959.16015625 disk.sda1.in‚0 disk.rbd0.out‚ 2.2421875 disk.sda1.out‚0 disk.sdb2.in‚5.40625 disk.rbd2.out‚ 2.466796875 disk.dm-0.in‚531338.54296875! disk.dm-0.out‚114062.92578125" proc.memory.rssMB‚ 3032.2265625) proc.memory.percent‚18.50724220275879 disk.rbd2.in‚0 disk.sdb1.in‚0 network.sent‚ 2.0227888e+07CJÁœ—"‚(:‘ ŸÝçÔ±²Ó‡ gpu.0.gpu‚42.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒúm˜"‚(:â ŸÝçÔƒ›õ† disk.rbd2.out‚ 2.466796875 disk.sdb.out‚ 992.177734375 disk.sda.in‚5.5625 disk.sda.out‚ 992.177734375! disk.dm-0.out‚114195.20703125* proc.memory.availableMB‚10120.73046875 disk.md0.in‚10.96875 disk.rbd2.in‚0) proc.memory.percent‚18.50724220275879$ disk./.usageGB‚2975.342441558838 disk.md0.out‚ 956.33203125" disk.nvme1n1.in‚201379.3828125# disk.nvme0n1.in‚329959.16015625 disk.sdb2.in‚5.40625 network.recv‚ 3.097849e+06 disk.sdb.in‚5.40625 disk.sdb2.out‚ 992.177734375" disk.nvme1n1.out‚ 59917.7421875 disk.sdb1.out‚0 disk.sda1.in‚0 disk.md0p1.in‚10.96875 disk.md0p1.out‚ 956.33203125# memory_percent‚38.2279634475708( disk./.usagePercent‚83.1918199435399 disk.rbd0.out‚ 2.2421875 disk.sda2.out‚ 992.177734375" proc.memory.rssMB‚ 3032.2265625 cpu‚25.191747649700453 network.sent‚ 2.0230977e+07 disk.rbd0.in‚0.015625# disk.nvme0n1.out‚54277.46484375 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.dm-0.in‚531338.54296875 disk.sda2.in‚5.5625@míÑ2™"- _wandbruntime‚9058  _runtime‚9058jJ¢ÌÆš"À & lr-AdamW/pg1‚0.0002835516846076106 & lr-AdamW/pg2‚0.0002835516846076106 trainer/global_step‚7749 % _timestamp‚1.7884730053144798e+09 _runtime‚ 9058.49949669 _step‚634zZn¹Ë›"Å _runtime‚ 9058.49949669 _step‚634 & lr-AdamW/pg1‚0.0002835516846076106 & lr-AdamW/pg2‚0.0002835516846076106 trainer/global_step‚7749 % _timestamp‚1.7884730053144798e+09úÜôF2œ"- _wandbruntime‚9058  _runtime‚9058äÍá_%"èO C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚331250928572928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 trainer/global_step‚7749 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚674 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 _runtime‚9058.500838366 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚315849345356800 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚334544634134016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15401583216128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚672 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚315893329428566 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15357922889830 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚331250928572928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 358233703 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 144109835 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚300535406538736 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 210775566 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚350206811703296 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 210775566 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 147458137 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚315849363359744 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88140016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55969821 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 % _timestamp‚1.7884730053156123e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 358207357 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 210749222 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _step‚635 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15401816884736 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147458135 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88140014 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚350206497143808 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚334544610193920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 147458135 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15401816884736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚315849363359744 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15662177569280 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚300535423578096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15661886949888 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚331251180244480 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 144110509 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55970493 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚¨Û 7582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚315893575449010 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0ÒLIló'ž"íO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚315893575449010 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚331250928572928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 trainer/global_step‚7749 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚674 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 _runtime‚9058.500838366 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚315849345356800 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚334544634134016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15401583216128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚672 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚315893329428566 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15357922889830 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚331250928572928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 358233703 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 144109835 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚300535406538736 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 210775566 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚350206811703296 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 210775566 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 147458137 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚315849363359744 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88140016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55969821 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 % _timestamp‚1.7884730053156123e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 358207357 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 210749222 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _step‚635 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15401816884736 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147458135 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88140014 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚350206497143808 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚334544610193920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 147458135 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15401816884736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚315849363359744 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15662177569280 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚300535423578096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15661886949888 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚331251180244480 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 144110509 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55970493 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582ûxˆ¿V2Ÿ"- _wandbruntime‚9059  _runtime‚9059®(@©÷& "ñM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280120320 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚300572953935520 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚315888631393792 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 144127474 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147476410 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚331292430320640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚315888649396736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 358278696 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15403780923904 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 304060416 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15403547241984 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚315888649396736 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15359881378816 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 _runtime‚9059.531794684 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147476412 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147476412 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚331292178635776 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147476410 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚350250181615616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15663887451648 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚315888631393792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88150816 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚334586318104064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚331292430320640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚659 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55976658 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 $ _timestamp‚1.788473006346224e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15360110360108 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 358252324 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 210775914 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88150818 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚315933064295628 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚315932818274976 trainer/global_step‚7749 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 210775914 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88150818 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚315933064295628 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚315932818274976 trainer/global_step‚7749 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚315888631393792 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 144127474 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147476410 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚331292430320640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚315888649396736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 358278696 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15403780923904 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 304060416 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15403547241984 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚315888649396736 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15359881378816 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 _runtime‚9059.531794684 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147476412 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147476412 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚331292178635776 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147476410 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚350250181615616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15663887451648 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚315888631393792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88150816 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚334586318104064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚331292430320640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚659 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55976658 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0ü<BZ2¢"- _wandbruntime‚9059  _runtime‚9059AòM_]£"× 2loss/SRqqt2/assignment_loss‚1.7219808803056367 2loss/SRqqt3/assignment_loss‚1.0051224451004104 2loss/FBt3/assignment_loss‚0.003929374632325562 .loss/FRt1/detection_loss‚3.000738338782237 .loss/FBt1/detection_loss‚2.402256604035695 0loss/FBt3/detection_loss‚0.49408470787728825 /loss/FBt4/detection_loss‚2.2778260665280476 /loss/FRt2/assignment_loss‚3.257101640556798 1loss/FBt4/assignment_loss‚0.20059885832485438 0loss/SRqqt1/detection_loss‚1.955163847582013 _step‚637 2loss/SRqqt1/assignment_loss‚1.4753684717066147 1loss/SRqqt4/detection_loss‚2.4800760698318483 /loss/FBt2/detection_loss‚2.3100741942723593 % _timestamp‚1.7884730063486328e+09 _runtime‚9059.533372231 /loss/FRt3/detection_loss‚1.5957517520634128 epoch‚6 0loss/FBt1/assignment_loss‚0.1503396417910608 1loss/FBt2/assignment_loss‚0.20093608870386864 0loss/SRqqt2/detection_loss‚2.365431345999241 /loss/FRt2/detection_loss‚2.4850653154179314 0loss/FRt1/assignment_loss‚2.8746086198550005 /loss/FRt3/assignment_loss‚1.070638937985196 0loss/FRt4/assignment_loss‚3.6973099267041243 .loss/FRt4/detection_loss‚3.242325007915497 &loss/total_loss‚42.945997248814464 1loss/SRqqt4/assignment_loss‚1.928647928237915 1loss/SRqqt3/detection_loss‚0.7506211846050891 trainer/global_step‚7749ˆ‘¬’¤"Ü /loss/FRt2/detection_loss‚2.4850653154179314 0loss/FRt1/assignment_loss‚2.8746086198550005 /loss/FRt3/assignment_loss‚1.070638937985196 0loss/FRt4/assignment_loss‚3.6973099267041243 .loss/FRt4/detection_loss‚3.242325007915497 &loss/total_loss‚42.945997248814464 1loss/SRqqt4/assignment_loss‚1.928647928237915 1loss/SRqqt3/detection_loss‚0.7506211846050891 trai7PCEÐner/global_step‚7749 2loss/SRqqt2/assignment_loss‚1.7219808803056367 2loss/SRqqt3/assignment_loss‚1.0051224451004104 2loss/FBt3/assignment_loss‚0.003929374632325562 .loss/FRt1/detection_loss‚3.000738338782237 .loss/FBt1/detection_loss‚2.402256604035695 0loss/FBt3/detection_loss‚0.49408470787728825 /loss/FBt4/detection_loss‚2.2778260665280476 /loss/FRt2/assignment_loss‚3.257101640556798 1loss/FBt4/assignment_loss‚0.20059885832485438 0loss/SRqqt1/detection_loss‚1.955163847582013 _step‚637 2loss/SRqqt1/assignment_loss‚1.4753684717066147 1loss/SRqqt4/detection_loss‚2.4800760698318483 /loss/FBt2/detection_loss‚2.3100741942723593 % _timestamp‚1.7884730063486328e+09 _runtime‚9059.533372231 /loss/FRt3/detection_loss‚1.5957517520634128 epoch‚6 0loss/FBt1/assignment_loss‚0.1503396417910608 1loss/FBt2/assignment_loss‚0.20093608870386864 0loss/SRqqt2/detection_loss‚2.365431345999241ý4÷é¥"‚(:„ ®ÝçÔÓŽ­† gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚114.74' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.685 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 _¸r¦"‚(:ç ®ÝçÔ„Ýì†( disk./.usagePercent‚83.1918332761064 disk.sda1.in‚0 disk.md0p1.in‚10.96875" proc.memory.rssMB‚ 3032.2265625# disk.nvme0n1.out‚54349.60546875) proc.memory.percent‚18.50724220275879 cpu‚25.191431738083306 disk.sdb1.out‚0 disk.rbd0.in‚0.015625" disk.nvme1n1.out‚ 60063.3203125 disk.sdb.out‚993.68115234375 disk.dm-0.in‚531338.54296875! disk.sda2.out‚993.68115234375 disk.sdb2.in‚5.40625! disk.sdb2.out‚993.68115234375 disk.rbd2.out‚ 2.466796875" disk.nvme1n1.in‚201379.3828125* proc.memory.availableMB‚10120.51171875 disk.sdb1.in‚0 disk.sda1.out‚0 disk.sda2.in‚5.5625 network.sent‚ 2.0237375e+07$ disk./.usageGB‚2975.342918395996 disk.sda.in‚5.5625 disk.sda.out‚993.68115234375! disk.dm-0.out‚114412.92578125 disk.md0.out‚ 957.765625 disk.rbd2.in‚0$ memory_percent‚38.22929859161377# disk.nvme0n1.in‚329959.16015625 disk.md0.in‚10.96875 disk.rbd0.out‚ 2.2421875 proc.cpu.threads‚17 network.recv‚ 3.102644e+06 disk.md0p1.out‚ 957.765625 disk.sdb.in‚5.40625S(ºŸ§"‚(:… ½ÝçÔñóÜ… gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚124.044' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.011 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>”Ïéq¨"‚(:æ ½ÝçÔ²õé‡ disk.sdb1.out‚0$ memory_percent‚38.22894096374512) proc.memory.availableMB‚ 10120.5703125 proc.cpu.threads‚17) disk./.usagePercent‚83.17708297779822 disk.sda2.in‚5.5625 disk.md0p1.in‚10.96875 disk.sdb2.out‚995.6435546875 disk.sdb.in‚5.40625! disk.dm-0.out‚114528.94921875 disk.sda1.in‚0 disk.md0.in‚10.96875 disk.sdb.out‚995.6435546875# disk.nvme0n1.in‚329959.16015625 disk.md0p1.out‚ 959.6796875 disk.rbd2.out‚ 2.478515625" disk.nvme1n1.in‚201379.3828125" proc.memory.rssMB‚ 3032.2265625 disk.dm-0.in‚531338.54296875 disk.sdb1.in‚0 disk.sda1.out‚0 disk.sdb2.in‚5.40625 disk.md0.out‚ 959.6796875 network.sent‚ 2.0347298e+07 network.recv‚ 3.110042e+06% disk./.usageGB‚2974.8153762817383 disk.sda2.out‚995.6435546875 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.sda.in‚5.5625 disk.sda.out‚995.6435546875# disk.nvme0n1.out‚54421.78515625 disk.rbd0.out‚ 2.3203125" disk.nvme1n1.out‚ 60107.1640625) proc.memory.percent‚18.50724220275879 cpu‚25.191193149695884ö<òq©"‚(:æ ÌÝçÔçž›‡ disk.sdb.in‚5.40625 disk.sdb.out‚997.15869140625 disk.sda.out‚997.15869140625 disk.md0p1.out‚ 961.140625# disk./.usageGB‚2974.86861038208" disk.nvme1n1.in‚201379.3828125 disk.sdb1.in‚0# disk.nvme0n1.in‚329959.16015625 disk.md0.in‚10.96875 disk.rbd2.out‚ 2.541015625 disk.sda.in‚5.5625! disk.sda2.out‚997.15869140625 network.sent‚ 2.0351097e+07 disk.rbd2.in‚0# disk.nvme1n1.out‚60107.83984375! disk.nvme0n1.out‚ 54493.859375$ memory_percent‚38.22743892669678" proc.memory.rssMB‚ 3032.2265625 disk.sdb1.out‚0 disk.sda2.in‚5.5625* proc.memory.availableMB‚10120.81640625 disk.dm-0.in‚531338.54296875! disk.dm-0.out‚114601.69921875 disk.sda1.in‚0 disk.sdb2.in‚5.40625 disk.md0.out‚ 961.140625 disk.rbd0.in‚0.015625 cpu‚25.190909785753867 network.recv‚ 3.114274e+06) disk./.usagePercent‚83.17857142552398 disk.md0p1.in‚10.96875! disk.sdb2.out‚997.15869140625) proc.memory.percent‚18.50724220275879 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd0.out‚ 2.3203125ef®‘ª"‚(:† ÌÝçÔ¶·â‡ gpu.0.gpu‚28.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚133.626' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.4065 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h­¸s«"‚(:è ÛÝçÔñü©‡ disk.sdb1.in‚0 disk.sdb1.out‚0 proc.cpu.threads‚17 disk.rbd0.out‚ 2.42236328125* proc.memory.availableMB‚10120.81640625) disk./.usagePercent‚83.17785552003238 disk.md0p1.in‚10.96875 disk.md0p1.out‚ 962.609375 disk.sdb.in‚5.40625 disk.sdb.out‚998.67822265625 network.sent‚ 2.0356402e+07# disk.nvme0n1.in‚329959.16015625 disk.dm-0.in‚531338.54296875 disk.sda2.in‚5.5625 disk.sdb2.in‚5.40625 disk.md0.in‚10.96875 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 2.541015625$ disk./.usageGB‚2974.843006134033! disk.dm-0.out‚114746.70703125 disk.rbd2.in‚0$ memory_percent‚38.22743892669678 cpu‚25.19062311273187 network.recv‚ 3.11794e+06 disk.sda.in‚5.5625 disk.sda1.in‚0 disk.sda1.out‚0! disk.sdb2.out‚998.67822265625" disk.nvme1n1.in‚201379.3828125 disk.sda.out‚998.67822265625" proc.memory.rssMB‚ 3032.2265625) proc.memory.percent‚18.50724220275879! disk.sda2.out‚998.67822265625 disk.md0.out‚ 962.609375 disk.nvme1n1.out‚ 60180.40625# disk.nvme0n1.out‚54566.30078125”—!b’¬"‚(:‡ ÛÝçÔýã㇠gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚139.251' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.81275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03\kL’­"‚(:‡ êÝçÔÅêч gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚138.919' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.72975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0É‹e%w®"‚(:ì êÝçÔÆóþ‡ disk.sdb1.in‚0" disk.sda2.out‚1000.75732421875 disk.md0p1.out‚ 964.6171875 disk.md0.in‚10.96875! disk.dm-0.out‚114898.47265625) disk./.usagePercent‚83.18134758585404" disk.sdb2.out‚1000.75732421875# disk.nvme1n1.out‚60260.12890625 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.dm-0.in‚531338.54296875 disk.sda1.in‚0$ memory_percent‚38.23051452636719" proc.memory.rssMB‚ 3032.2265625 cpu‚25.190361508507127 disk.rbd2.out‚ 2.6201171875# disk.nvme0n1.in‚329959.16015625 disk.nvme0n1.out‚ 54638.34375 network.recv‚ 3.122228e+06 disk.sdb.in‚5.40625! disk.sdb.out‚1000.75732421875! disk.sda.out‚1000.75732421875) proc.memory.percent‚18.50724220275879 disk.md0.out‚ 964.6171875 disk.rbd0.in‚0.015625" disk.nvme1n1.in‚201379.3828125& proc.memory.availableMB‚ 10120.3125% disk./.usageGB‚2974.9678993225098 disk.sda2.in‚5.5625 disk.md0p1.in‚10.96875 disk.rbd0.out‚ 2.42236328125 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sdb2.in‚5.40625 network.sent‚ 2.03649e+07 disk.sda.in‚5.5625†¡"•2¯"- _wandbruntime‚9120  _runtime‚9120ΘBÅ°"¿ ' lr-AdamW/pg2‚0.00027791167571612215 trainer/global_step‚7799 # _timestamp‚1.78847306748955e+09 _runtime‚ 9120.6788197 _step‚638 ' lr-AdamW/pg1‚0.00027791167571612215äçrʱ"Ä ' lr-AdamW/pg1‚0.00027791167571612215 ' lr-AdamW/pg2‚0.00027791167571612215 trainer/global_step‚7799 # _timestamp‚1.78847306748955e+09 _runtime‚ 9120.6788197 _step‚638þy‰Ñs2²"- _wandbruntime‚9120  _runtime‚9120Ï‚ýfî'³"èO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚333313446129152 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚317813647206400 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚691 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56317680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7884730674949448e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚693 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚352390903626752 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15456089911422 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 212087174 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15455860930338 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚333313697800704 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _step‚639 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚7799 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚333313697800704 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 144998487 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚317868031360718 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 _runtime‚9120.681186471 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 212113518 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚302411924409936 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 360459059 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15500032591360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15499798922752 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 212087174 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88680116 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚317813647206400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15499798922752 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15500032591360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚336628808696320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15762070990336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 148371887 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88680114 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56318371 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15761778273792 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚317813665209344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚317813665209344 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚333313446129152 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148371885 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309‘ìðó'´"íO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚317813665209344 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚333313446129152 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148371885 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚333313446129152 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚317813647206400 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚691 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56317680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7884730674949448e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚693 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚352390903626752 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15456089911422 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 212087174 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15455860930338 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚333313697800704 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _step‚639 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚7799 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚333313697800704 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 144998487 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚317868031360718 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 _runtime‚9120.681186471 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 212113518 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚302411924409936 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 360459059 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15500032591360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15499798922752 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 212087174 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88680116 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚317813647206400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15499798922752 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15500032591360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚336628808696320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15762070990336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 148371887 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88680114 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56318371 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15761778273792 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚317813665209344ÿ"_{M2µ"- _wandbruntime‚9122  _runtime‚9122‘,N“ ¶"òM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚317852951246336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56325520 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15457819419324 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 148390160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56324832 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚317907520207336 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚302449454767360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88690918 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚317907274186684 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15763765887488 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15501762948608 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚302449471806720 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _step‚640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚317852933243392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 M>DeviceStatsMonitor.on_train%8 e_batch_end/requested_bytes.all.peak‚ 6741059457 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15458048400616 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 360530398 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 145016438 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚9122.201168873 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚333354947876864 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚333354947876864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚333354696192000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚336670516606464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 212140236 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 212113866 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15764060687872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 212140236 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚317852951246336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145015748 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 360504026 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 360530398 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582rƒ…Cý&·"÷M D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15458048400616 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 360530398 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 145016438 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚9122.201168873 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚333354947876864 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚333354947876864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚333354696192000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚336670516606464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 212140236 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 212113866 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15764060687872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 212140236 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚317852951246336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145015748 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 360504026 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 360530398 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚317852951246336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56325520 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15457819419324 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 148390160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56324832 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚317907520207336 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚302449454767360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88690918 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚317907274186684 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15763765887488 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15501762948608 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚302449471806720 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _step‚640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚317852933243392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608€ «732¸"- _wandbruntime‚9122  _runtime‚9122&Ón]¹"× 2loss/SRqqt4/assignment_loss‚1.3115271904638834 2loss/FBt3/assignment_loss‚0.053090304184138404 /loss/FRt1/detection_loss‚2.5757801458239555 .loss/FRt4/detection_loss‚2.826817917101311 0loss/SRqqt2/detection_loss‚2.512211256027222 % _timestamp‚1.7884730690175347e+09 0loss/FRt3/assignment_loss‚1.2915799554550287 /loss/FBt1/detection_loss‚2.1567514809695156 .loss/FBt4/detection_loss‚2.783557193619864 %loss/total_loss‚40.66568036958756 epoch‚6 _runtime‚9122.202206807 _step‚641 0loss/FRt1/assignment_loss‚2.3735751271247865 0loss/FBt1/assignment_loss‚0.0885490833962463 1loss/FBt2/assignment_loss‚0.07679982265938179 trainer/global_step‚7799 2loss/SRqqt2/assignment_loss‚1.7514528150856494 1loss/FBt4/assignment_loss‚0.36032292822242845 1loss/SRqqt3/detection_loss‚0.7868992389437366 /loss/FRt3/detection_loss‚1.4176459437179747 1loss/SRqqt1/detection_loss‚1.7515436313590225 .loss/FRt2/assignment_loss‚2.99486064016819 /loss/FRt2/detection_loss‚2.2842603661702015 1loss/SRqqt4/detection_loss‚2.7069504175867354 /loss/FRt4/assignment_loss‚3.609468521493854 /loss/FBt2/detection_loss‚1.8063011765480042 /loss/FBt3/detection_loss‚0.5290036627261991 2loss/SRqqt1/assignment_loss‚1.5426258417416592 2loss/SRqqt3/assignment_loss‚1.0741057089985686R½ç bº"Ü .loss/FRt2/assignment_loss‚2.99486064016819 /loss/FRt2/detection_loss‚2.2842603661702015 1loss/SRqqt4/detection_loss‚2.7069504175867354 /loss/FRt4/assignment_loss‚3.609468521493854 /loss/FBt2/detection_loss‚1.8063011765480042 /loss/FBt3/detection_loss‚0.5290036627261991 2loss/SRqqt1/assignment_loss‚1.5426258417416592 2loss/SRqqt3/assignment_loss‚1.0741057089985686 2loss/SRqqt4/assignment_loss‚1.3115271904638834 2loss/FBt3/assignment_loss‚0.053090304184138404 /loss/FRt1/detection_loss‚2.5757801458239555 .loss/FRt4/detection_loss‚2.826817917101311 0loss/SRqqt2/detection_loss‚2.512211256027222 % _timestamp‚1.7884730690175347e+09 0loss/FRt3/assignment_loss‚1.2915799554550287 /loss/FBt1/detection_loss‚2.1567514809695156 .loss/FBt4/detection_loss‚2.783557193619864 %loss/total_loss‚40.66568036958756 epoch‚6 _runtime‚9122.202206807 _step‚641 0loss/FRt1/assignment_loss‚2.3735751271247865 0loss/FBt1/assignment_loss‚0.0885490833962463 1loss/FBt2/assignment_loss‚0.07679982265938179 trainer/global_step‚7799 2loss/SRqqt2/assignment_loss‚1.7514528150856494 1loss/FBt4/assignment_loss‚0.36032292822242845 1loss/SRqqt3/detection_loss‚0.7868992389437366 /loss/FRt3/detection_loss‚1.4176459437179747 1loss/SRqqt1/detection_loss‚1.7515436313590225Æ&Úœ»"‚(:‘ ùÝçÔ¼½ã‡ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚119.368' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.841999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨t±py¼"‚(:î ùÝçԞ뵈 disk.md0.out‚ 965.953125" disk.nvme1n1.in‚201379.3828125 cpu‚25.19010212606167$ memory_percent‚38.22922706604004) proc.memory.availableMB‚ 10120.5234375) disk./.usagePercent‚83.15515751883369# disk.nvme0n1.in‚329959.16015625# disk.nvme0n1.out‚54759.76953125 disk.sdb.in‚5.40625 disk.sdb2.in‚5.40625 proc.cpu.threads‚17% disk./.usageGB‚2974.0312156677246 disk.sdb1.out‚0 disk.sda1.out‚0" disk.sda2.out‚1002.14697265625 disk.md0p1.out‚ 965.953125 disk.rbd0.in‚0.015625# disk.nvme1n1.out‚60320.38671875 disk.md0p1.in‚10.96875 disk.md0.in‚10.96875 disk.rbd2.out‚ 2.6201171875) proc.memory.percent‚18.50724220275879 network.sent‚ 2.0367584e+07! disk.sda.out‚1002.14697265625 disk.dm-0.in‚531338.54296875 disk.rbd2.in‚0 network.recv‚ 3.125827e+06! disk.sdb.out‚1002.14697265625 disk.sda.in‚5.5625 disk.sda1.in‚0 disk.sda2.in‚5.5625 disk.rbd0.out‚ 2.47021484375" proc.memory.rssMB‚ 3032.2265625 disk.sdb1.in‚0 disk.dm-0.out‚ 115080.15625" disk.sdb2.out‚1002.14697265625+ æؽ"‚(:… ˆÞçÔܺ‡ gpu.0.gpu‚27.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚130.232' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.558 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥œe8o¾"‚(:ä ˆÞçÔ¶ôè‡$ memory_percent‚38.22767734527588 cpu‚25.189822294648895% disk./.usageGB‚2974.2822341918945 disk.rbd0.in‚0.015625 disk.sdb.out‚ 1003.64453125# disk.nvme0n1.in‚329959.16015625 disk.rbd2.out‚ 2.68310546875 disk.sda2.out‚ 1003.64453125 disk.md0p1.in‚10.96875( disk./.usagePercent‚83.1621761018289 disk.rbd0.out‚ 2.47021484375 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.nvme0n1.out‚ 54759.84375 disk.md0p1.out‚ 967.39453125 disk.sdb2.in‚5.40625 disk.sdb1.in‚0 disk.sda.in‚5.5625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb.in‚5.40625 disk.dm-0.in‚531338.54296875 disk.sdb2.out‚ 1003.64453125 disk.md0.out‚ 967.39453125 proc.cpu.threads‚17 disk.sda2.in‚5.5625 disk.md0.in‚10.96875* proc.memory.availableMB‚10120.77734375" proc.memory.rssMB‚ 3032.2265625) proc.memory.percent‚18.50724220275879" disk.nvme1n1.in‚201379.3828125 disk.dm-0.out‚ 115209.421875 network.sent‚ 2.0477576e+07 network.recv‚ 3.133125e+06! disk.nvme1n1.out‚ 60449.578125 disk.sda.out‚ 1003.64453125…¶’¿"‚(:‡ —ÞçÔÆæ’‡ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚136.743' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.18575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!õFcpÀ"‚(:å —ÞçÔäªá‡ disk.md0p1.out‚ 969.359375$ memory_percent‚38.22767734527588) proc.memory.percent‚18.50724220275879) disk./.usagePercent‚83.16215188988811 disk.sdb.out‚1005.658203125 proc.cpu.threads‚17 network.sent‚ 2.0482836e+07 disk.rbd0.out‚ 2.533203125 disk.rbd2.out‚ 2.68310546875 disk.sdb.in‚5.40625 disk.sdb1.in‚0 disk.sda.in‚5.5625 disk.sda2.in‚5.5625 network.recv‚ 3.13696e+06 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0% disk./.usageGB‚2974.2813682556152 disk.sdb2.in‚5.40625 disk.md0.out‚ 969.359375# disk.nvme0n1.in‚329959.16015625! disk.nvme0n1.out‚ 54760.140625 disk.dm-0.out‚115347.4140625 disk.sdb2.out‚1005.658203125 disk.rbd0.in‚0.015625" disk.nvme1n1.in‚201379.3828125" disk.nvme1n1.out‚ 60587.2734375 disk.sdb1.out‚0 disk.sda2.out‚1005.658203125 cpu‚25.189567448865798 disk.md0.in‚10.96875 disk.sda.out‚1005.658203125 disk.dm-0.in‚531338.54296875 disk.md0p1.in‚10.96875* proc.memory.availableMB‚10120.77734375" proc.memory.rssMB‚ 3032.2265625LŸ¡¦2Á"- _wandbruntime‚9171  _runtime‚9171ÿ­ÉÂ"à ' lr-AdamW/pg1‚0.00027229965780166214 ' lr-AdamW/pg2‚0.00027229965780166214 trainer/global_step‚7849 % _timestamp‚1.7884731180819008e+09 _runtime‚9171.266891275 _step‚642Ì’ÎÃ"È % _timestamp‚1.7884731180819008e+09 _runtime‚9171.266891275 _step‚642 ' lr-AdamW/pg1‚0.00027229965780166214 ' lr-AdamW/pg2‚0.00027229965780166214 trainer/global_step‚7849‚Ð y2Ä"- _wandbruntime‚9171  _runtime‚9171hÓX,Å"èO WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 213425126 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56665959 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚686 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚335375963685376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚338713031138816 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15861950140416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 213425126 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89220214 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚335376215356928 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚338713007198720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚319777949056000 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 213451470 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚319842241251982 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56665273 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚354574981279232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚9171.268231891 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚335375963685376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requestlï4Ç ed_bytes.small_pool.peak‚ 560288777 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15554027951930 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 362710761 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15598014629376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 149285635 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89220216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚319842487272426 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149285637 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15553798970846 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084flÏÅó'Æ"íO XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 213425126 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56665959 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚686 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚335375963685376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚338713031138816 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15861950140416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 213425126 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89220214 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚335376215356928 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚338713007198720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚319777949056000 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 213451470 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚319842241251982 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56665273 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚354574981279232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚9171.268231891 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚335375963685376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15554027951930 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 362710761 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15598014629376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 149285635 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89220216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚319842487272426 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149285637 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15553798970846 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712ƒÔm32Ç"- _wandbruntime‚9172  _runtime‚9172d­Wø&È"òM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15599978655232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚354618359780864 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89231018 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚354618670132736 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15555757459832 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 213478188 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚338754691168768 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7884731190595162e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149303910 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15600212337152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _runtime‚9172.245286691 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚7849 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚319817235092992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89231016 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 362782100 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚335417213748224 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149303910 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚319817253095936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚304325989677920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145903239 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 213451818 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 213451818 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 362755728 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚319817235092992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚335417465433088 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15600212337152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56672860 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚338754715108864 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15555986441124 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚644 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚319817253095936 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15863955023872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚319881730098392 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚304325972638560 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149303912 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15599978655232 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 362755728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56672223 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 362782100 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15863668612096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚319881976119044 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 5725224969J2Rý&É"÷M B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚319817253095936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚304325989677920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145903239 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 213451818 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 213451818 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 362755728 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚319817235092992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚335417465433088 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15600212337152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56672860 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚338754715108864 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15555986441124 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚644 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚319817253095936 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15863955023872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚319881730098392 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚304325972638560 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149303912 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15599978655232 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 362755728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56672223 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 362782100 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15863668612096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚319881976119044 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15599978655232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚354618359780864 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89231018 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚354618670132736 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15555757459832 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 213478188 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚338754691168768 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7884731190595162e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149303910 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15600212337152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _runtime‚9172.245286691 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚7849 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚319817235092992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89231016 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 362782100 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚335417213748224 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149303910 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0„V™{2Ê"- _wandbruntime‚9172  _runtime‚9172Ž»Š~_Ë"Ù /loss/FRt4/assignment_loss‚3.515472589267625 2loss/SRqqt4/assignment_loss‚1.4227109527629282 2loss/FBt1/assignment_loss‚0.012023992193159453 0loss/FBt3/assignment_loss‚0.0413971923926973 _runtime‚9172.246704508 1loss/SRqqt2/assignment_loss‚1.345526017135743 0loss/FBt4/assignment_loss‚0.2952357393222782 _step‚645 0loss/FRt2/assignment_loss‚3.0449033956974745 /loss/FRt3/detection_loss‚1.3491769882643387 0loss/FRt1/assignment_loss‚2.6930634654485264 2loss/SRqqt3/assignment_loss‚0.8747120500379002 /loss/FRt1/detection_loss‚2.3998386217997623 % _timestamp‚1.7884731190619466e+09 /loss/FRt4/detection_loss‚2.7125192979971566 1loss/SRqqt1/detection_loss‚1.9323294103145598 /loss/FRt2/detection_loss‚2.4781624364150048 0loss/SRqqt4/detection_loss‚2.363567194177045 /loss/FBt1/detection_loss‚2.4423800773090787 /loss/FBt3/detection_loss‚0.7838404046576786 /loss/FBt4/detection_loss‚3.0727685689926147 %loss/total_loss‚41.03218636550737 0loss/FRt3/assignment_loss‚1.2801406565639708 1loss/FBt2/assignment_loss‚0.06005325632064995 1loss/SRqqt2/detection_loss‚2.4212225151784494 /loss/FBt2/detection_loss‚2.0042125303298235 trainer/global_step‚7849 2loss/SRqqt1/assignment_loss‚1.5299926451510852 1loss/SRqqt3/detection_loss‚0.9569363677778195 epoch‚6½­¨dÌ"Þ 0loss/FRt2/assignment_loss‚3.0449033956974745 /loss/FRt3/detection_loss‚1.3491769882643387 0loss/FRt1/assignment_loss‚2.6930634654485264 2loss/SRqqt3/assignment_loss‚0.8747120500379002 /loss/FRt1/detection_loss‚2.3998386217997623 % _timestamp‚1.7884731190619466e+09 /loss/FRt4/detection_loss‚2.7125192979971566 1loss/SRqqt1/detection_loss‚1.9323294103145598 /loss/FRt2/detection_loss‚2.4781624364150048 0loss/SRqqt4/detection_loss‚2.363567194177045 /loss/FBt1/detection_loss‚2.4423800773090787 /loss/FBt3/detection_loss‚0.7838404046576786 /loss/FBt4/detection_loss‚3.0727685689926147 %loss/total_loss‚41.03218636550737 0loss/FRt3/assignment_loss‚1.2801406565639708 1loss/FBt2/assignment_loss‚0.06005325632064995 1loss/SRqqt2/detection_loss‚2.4212225151784494 /loss/FBt2/detection_loss‚2.0042125303298235 trainer/global_step‚7849 2loss/SRqqt1/assignment_loss‚1.5299926451510852 1loss/SRqqt3/detection_loss‚0.9569363677778195 epoch‚6 /loss/FRt4/assignment_loss‚3.515472589267625 2loss/SRqqt4/assignment_loss‚1.4227109527629282 2loss/FBt1/assignment_loss‚0.012023992193159453 0loss/FBt3/assignment_loss‚0.0413971923926973 _runtime‚9172.246704508 1loss/SRqqt2/assignment_loss‚1.345526017135743 0loss/FBt4/assignment_loss‚0.2952357393222782 _step‚645…(Ñ‘Í"‚(:† ¦ÞçÔÛÚ‡ gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚115.966' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.9915 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£…ϯqÎ"‚(:æ ¦ÞçÔ­Àƈ" disk.nvme1n1.out‚ 60699.7578125 cpu‚25.18930855164448 disk.sda2.out‚1007.228515625$ disk./.usageGB‚2974.281421661377 disk.sdb2.in‚5.40625 disk.sdb2.out‚1007.228515625 disk.rbd2.in‚0 disk.sdb.out‚1007.228515625 disk.sdb1.out‚0 network.sent‚ 2.0499754e+07" disk.nvme1n1.in‚201379.3828125 disk.md0p1.out‚ 970.8671875* proc.memory.availableMB‚10120.20703125# disk.nvme0n1.out‚54760.26953125 disk.sda1.in‚0$ memory_percent‚38.23115825653076" proc.memory.rssMB‚ 3032.2265625 proc.cpu.threads‚17 disk.sda.in‚5.5625 disk.dm-0.in‚531338.54296875 disk.sda2.in‚5.5625 network.recv‚ 3.141402e+06) disk./.usagePercent‚83.16215338313555) proc.memory.percent‚18.50724220275879 disk.md0.out‚ 970.8671875 disk.rbd0.in‚0.015625 disk.sdb1.in‚0 disk.sda.out‚1007.228515625# disk.nvme0n1.in‚329959.16015625 disk.sda1.out‚0 disk.md0.in‚10.96875 disk.rbd2.out‚ 2.74609375 disk.sdb.in‚5.40625! disk.dm-0.out‚115460.02734375 disk.md0p1.in‚10.96875 disk.rbd0.out‚ 2.533203125’H Ö‘Ï"‚(:† µÞçÔ”í‡ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚100.482' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.1205 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y¢?}Ð"‚(:ò µÞçÔÔˆÛ‡ disk.rbd2.out‚ 2.74609375 disk.sdb2.in‚5.40625" disk.sdb2.out‚1008.71435546875 disk.sdb1.out‚0 cpu‚25.189032770098798) disk./.usagePercent‚83.14740425809323 disk.sda1.in‚0 disk.md0p1.out‚ 972.28515625) proc.memory.percent‚18.50724220275879 proc.cpu.threads‚17 network.recv‚ 3.148533e+06 disk.md0.out‚ 972.28515625! disk.sdb.out‚1008.71435546875 disk.sda.in‚5.5625 disk.md0p1.in‚10.96875$ memory_percent‚38.22808265686035 disk.rbd2.in‚0 disk.dm-0.out‚115504.0546875# disk.nvme0n1.in‚329959.16015625# disk.nvme0n1.out‚54760.64453125 disk.dm-0.in‚531338.54296875" disk.nvme1n1.in‚201379.3828125 network.sent‚ 2.0594303e+07$ disk./.usageGB‚2973.753921508789 disk.rbd0.out‚ 2.59619140625) proc.memory.availableMB‚ 10120.7109375 disk.sda1.out‚0 disk.sda2.in‚5.5625" disk.sda2.out‚1008.71435546875" proc.memory.rssMB‚ 3032.2265625 disk.md0.in‚10.96875 disk.sdb1.in‚0 disk.rbd0.in‚0.015625# disk.nvme1n1.out‚60743.41015625 disk.sdb.in‚5.40625! disk.sda.out‚1008.71435546875ú´.ª|Ñ"‚(:ñ ÄÞçÔïôLJ disk.md0.out‚ 973.69140625! disk.dm-0.out‚115506.19921875 disk.rbd0.in‚0.015625 disk.sdb.in‚5.40625 disk.dm-0.in‚531338.54296875$ memory_percent‚38.22808265686035 network.sent‚ 2.0602533e+07 disk.md0p1.out‚ 973.69140625 disk.rbd2.in‚0 proc.cpu.threads‚17( disk./.usagePercent‚83.1426538113136 disk.sda1.in‚0# disk.nvme0n1.out‚54762.08203125 cpu‚25.188753882634355 disk.sda2.in‚5.5625 disk.md0p1.in‚10.96875 disk.md0.in‚10.96875 disk.rbd2.out‚ 2.74609375" disk.nvme1n1.in‚201379.3828125 disk.sdb1.in‚0 disk.sda.in‚5.5625 network.recv‚ 3.15274e+06% disk./.usageGB‚2973.5840225219727 disk.sda1.out‚0" disk.sda2.out‚1010.17724609375 disk.sdb2.in‚5.40625! disk.sdb.out‚1010.17724609375) proc.memory.percent‚18.50724220275879 disk.rbd0.out‚ 2.59619140625" disk.nvme1n1.out‚ 60744.1171875 disk.sdb1.out‚0! disk.sda.out‚1010.17724609375# disk.nvme0n1.in‚329959.16015625) proc.memory.availableMB‚ 10120.7109375" disk.sdb2.out‚1010.17724609375" proc.memory.rssMB‚ 3032.2265625q,½Ò"‚(:… ÄÞçÔ²¾Ò‡ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚120.31' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.0775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 à'"œÓ"‚(:‘ ÓÞçÔÇ“‡ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚119.368' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.841999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DAÈÂuÔ"‚(:ê ÓÞçÔõâȇ network.recv‚ 3.156648e+06) disk./.usagePercent‚83.14752382454974 disk.sdb1.in‚0# disk.nvme0n1.in‚329959.16015625# disk.nvme0n1.out‚54762.35546875 disk.sda2.in‚5.5625 disk.sda2.out‚1012.326171875 disk.md0p1.out‚ 975.76953125 disk.sdb2.in‚5.40625" disk.nvme1n1.in‚201379.3828125 disk.sdb.in‚5.40625" proc.memory.rssMB‚ 3032.2265625 disk.md0p1.in‚10.96875 disk.rbd2.in‚0 disk.sdb.out‚1012.326171875$ memory_percent‚38.22808265686035 proc.cpu.threads‚17 network.sent‚ 2.0605629e+07 disk.sda.in‚5.5625 disk.sda.out‚1012.326171875! disk.dm-0.out‚115578.59765625 disk.sda1.in‚0 cpu‚25.188475057635504$ disk./.usageGB‚2973.758197784424 disk.sda1.out‚0 disk.md0.out‚ 975.76953125 disk.rbd2.out‚ 2.74609375) proc.memory.percent‚18.50724220275879 disk.dm-0.in‚531338.54296875 disk.sdb2.out‚1012.326171875 disk.md0.in‚10.96875) proc.memory.availableMB‚ 10120.7109375 disk.sdb1.out‚0 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 2.59619140625" disk.nvme1n1.out‚ 60816.2421875÷¦§Ú2Õ"- _wandbruntime‚9226  _runtime‚9226Ĩ HÇÖ"Á trainer/global_step‚7899 % _timestamp‚1.7884731728371904e+09 _runtime‚9226.022079754 _step‚646 & lr-AdamW/pg1‚0.0002667168004003526 & lr-AdamW/pg2‚0.0002667168004003526:·ÐÌ×"Æ & lr-AdamW/pg1‚0.0002667168004003526 & lr-AdamW/pg2‚0.0002667168004003526 trainer/global_step‚7899 % _timestamp‚1.7884731728371904e+09 _runtime‚9226.022079754 _step‚646†uRë¤2Ø"- _wandbruntime‚9226  _runtime‚9226Å® Ëî'Ù"èO QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚337438481241600 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 trainer/global_step‚7899 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚321742268908544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15696230336000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89760314 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 214789422 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15696230336000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚337438481241600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15961821045248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 146776490 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 214763078 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚699 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 364988809 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚697 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15961528328704 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚356758734029824 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚321742268908544 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 150199387 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884731728381472e+09 _runtime‚9226.023132189 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚337438732913152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 J”,ó'Ú"íO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 150199387 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚321742250905600 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚340797229641216 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15696230336000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚337438481241600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15961821045248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 146776490 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 214763078 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚699 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 364988809 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚697 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15961528328704 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚356758734029824 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚321742268908544 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 150199387 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884731728381472e+09 _runtime‚9226.023132189 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚337438732913152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚337438481241600 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 trainer/global_step‚7899 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚321742268908544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15696230336000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89760314 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 214789422 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 214789770 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15698194361856 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚356802725926912 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 365033802 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚356802407186432 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚340838889671168 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15653924481632 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H@DeviceStatsMonitóë­F©or.on_train_batch_end/segment.small_pool.current‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚337479982989312 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 146794803 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 150217662 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884731738156605e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89771118 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚321781554945536 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 214816140 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 365033802 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 214816140 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57023685 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚306202507549120 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚321781536942592 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15698428043776 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚321856432030752 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 214789770 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15963517515264 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15653695500340 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 365007430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚337479731304448 trainer/global_step‚7899 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 365007430 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚648 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚321781536942592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89771116 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚721 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚306202490509760 _runtime‚9227.001003419 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚340838913611264 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15963812315648 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚321856186010100 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57022964 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15698428043776 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 150217662 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚321781554945536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146794080²½ý&Ý"÷M S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚337479731304448 trainer/global_step‚7899 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 365007430 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚648 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚321781536942592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89771116 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚721 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚306202490509760 _runtime‚9227.001003419 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚340838913611264 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15963812315648 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚321856186010100 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57022964 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15698428043776 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 150217662 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚321781554945536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146794080 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 214789770 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15698194361856 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚356802725926912 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 365033802 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚356802407186432 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚340838889671168 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15653924481632 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚337479982989312 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 146794803 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 150217662 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884731738156605e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89771118 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚321781554945536 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 214816140 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 365033802 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 214816140 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57023685 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚306202507549120 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚321781536942592 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15698428043776 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚321856432030752 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 214789770 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15963517515264 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15653695500340 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 365007430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273ˆM·¹#2Þ"- _wandbruntime‚9227  _runtime‚9227Å-è­`ß"Ú /loss/FRt4/assignment_loss‚2.775938416680982 /loss/FRt1/detection_loss‚2.4914575301938586 /loss/FRt3/detection_loss‚1.4116325557909228 .loss/FRt4/detection_loss‚2.734746963747086 1loss/SRqqt1/detection_loss‚1.9763334477847476 /loss/FBt3/detection_loss‚0.7756625510272429 %loss/total_loss‚41.14633475943708 % _timestamp‚1.7884731738178802e+09 _step‚649 0loss/FRt2/assignment_loss‚3.9214075880172925 2loss/SRqqt3/assignment_loss‚0.9287871241702725 3loss/FBt2/assignment_loss‚0.0009009648740526576 1loss/FBt4/assignment_loss‚0.22205546032591883 /loss/FRt2/detection_loss‚2.2495811244783304 0loss/SRqqt2/detection_loss‚2.107079697282691 /loss/FBt1/detection_loss‚2.6435376670625477 _runtime‚9227.002607837 2loss/SRqqt4/assignment_loss‚1.7302341949984883 1loss/FBt3/assignment_loss‚0.10125963848531253 trainer/global_step‚7899 0loss/FRt1/assignment_loss‚2.6580254236857095 1loss/FBt1/assignment_loss‚0.07196904960536206 1loss/SRqqt3/detection_loss‚0.8107242855386665 /loss/FBt4/detection_loss‚2.5973668158054353 2loss/SRqqt1/assignment_loss‚1.5426552646863714 2loss/SRqqt2/assignment_loss‚1.5928724314037122 epoch‚6 0loss/FRt3/assignment_loss‚1.3155233869046876 0loss/SRqqt4/detection_loss‚2.386148765231624 .loss/FBt2/detection_loss‚2.100434411655773íÞÌ eà"ß 2loss/SRqqt3/assignment_loss‚0.9287871241702725 3loss/FBt2/assignment_loss‚0.0009009648740526576 1loss/FBt4/assignment_loss‚0.22205546032591883 /loss/FRt2/detection_loss‚2.2495811244783304 0loss/SRqqt2/detection_loss‚2.107079697282691 /loss/FBt1/detection_loss‚2.6435376670625477 _runtime‚9227.002607837 2loss/SRqqt4/assignment_loss‚1.7302341949984883 1loss/FBt3/assignment_loss‚0.10125963848531253 trainer/global_step‚7899 0loss/FRt1/assignment_loss‚2.6580254236857095 1loss/FBt1/assignment_loss‚0.07196904960536206 1loss/SRqqt3/detection_loss‚0.8107242855386665 /loss/FBt4/detection_loss‚2.5973668158054353 2loss/SRqqt1/assignment_loss‚1.5426552646863714 2loss/SRqqt2/assignment_loss‚1.5928724314037122 epoch‚6 0loss/FRt3/assignment_loss‚1.3155233869046876 0loss/SRqqt4/detection_loss‚2.386148765231624 .loss/FBt2/detection_loss‚2.100434411655773 /loss/FRt4/assignment_loss‚2.775938416680982 /loss/FRt1/detection_loss‚2.4914575301938586 /loss/FRt3/detection_loss‚1.4116325557909228 .loss/FRt4/detection_loss‚2.734746963747086 1loss/SRqqt1/detection_loss‚1.9763334477847476 /loss/FBt3/detection_loss‚0.7756625510272429 %loss/total_loss‚41.14633475943708 % _timestamp‚1.7884731738178802e+09 _step‚649 0loss/FRt2/assignment_loss‚3.9214075880172925‰) ÷á"‚(:… âÞçԃȆ‡ gpu.0.gpu‚42.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚143.872' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.968 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´Èëþtâ"‚(:é âÞçÔ£²Ä‡ disk.sdb2.in‚5.40625 disk.rbd0.out‚ 2.59619140625 disk.rbd2.in‚0" disk.nvme1n1.out‚ 60870.1640625 disk.sdb1.out‚0# disk.nvme0n1.in‚329959.16015625 disk.nvme0n1.out‚54947.75 disk.md0p1.in‚10.96875 disk.md0.out‚ 977.3515625 disk.md0p1.out‚ 977.3515625) proc.memory.percent‚18.50724220275879" disk.sdb2.out‚1013.98095703125 disk.md0.in‚10.96875! disk.sdb.out‚1013.98095703125 disk.dm-0.out‚115817.9140625 disk.sda.in‚5.5625* proc.memory.availableMB‚10120.44140625" proc.memory.rssMB‚ 3032.2265625 disk.rbd2.out‚ 2.74609375 disk.sdb.in‚5.40625! disk.sda.out‚1013.98095703125 disk.sda1.out‚0 cpu‚25.18819732681997) disk./.usagePercent‚83.15323741593443 disk.dm-0.in‚531338.54296875" disk.sda2.out‚1013.98095703125 network.sent‚ 2.0612024e+07 disk.rbd0.in‚0.015625" disk.nvme1n1.in‚201379.3828125 disk.sdb1.in‚0 disk.sda2.in‚5.5625 proc.cpu.threads‚17 network.recv‚ 3.161013e+06$ disk./.usageGB‚2973.962543487549 disk.sda1.in‚0$ memory_percent‚38.22972774505615wÛÄ’ã"‚(:‡ ñÞçԊІ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚132.399' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.09975 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îTKÓ|ä"‚(:ñ ñÞçÔËêë† disk.sdb1.out‚0 network.sent‚ 2.071946e+07$ disk./.usageGB‚2974.087333679199 disk.sdb2.in‚5.40625 disk.md0.in‚10.96875 disk.sdb.in‚5.40625) proc.memory.percent‚18.50724220275879 proc.cpu.threads‚17! disk.dm-0.out‚115947.75390625 disk.sda1.out‚0! disk.sda.out‚1015.40869140625" disk.sdb2.out‚1015.40869140625 disk.rbd2.in‚0 disk.rbd2.out‚ 2.74609375$ memory_percent‚38.22979927062988" proc.memory.rssMB‚ 3032.2265625 disk.rbd0.out‚ 2.59619140625# disk.nvme1n1.out‚60999.58984375 disk.sda.in‚5.5625" disk.nvme0n1.out‚ 54948.1640625 disk.rbd0.in‚0.015625" disk.nvme1n1.in‚201379.3828125 disk.sdb1.in‚0) proc.memory.availableMB‚ 10120.4296875 disk.md0p1.out‚ 978.73046875 disk.md0.out‚ 978.73046875 cpu‚25.187924248872715 network.recv‚ 3.168223e+06# disk.nvme0n1.in‚329959.16015625 disk.sda2.in‚5.5625" disk.sda2.out‚1015.40869140625 disk.md0p1.in‚10.96875) disk./.usagePercent‚83.15672660192173 disk.dm-0.in‚531338.54296875 disk.sda1.in‚0! disk.sdb.out‚1015.40869140625ÀZ¨‘å"‚(:† €ßçÔ¤•ˆ‡ gpu.0.gpu‚25.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚116.306' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.0765 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â·wæ"‚(:ì €ßçÔ¸¬‡ disk.sdb.out‚1017.3173828125$ memory_percent‚38.22982311248779) proc.memory.percent‚18.50724220275879 cpu‚25.187618382945953 network.recv‚ 3.172627e+06 disk.rbd2.in‚0 disk.sda2.in‚5.5625 disk.sdb2.in‚5.40625" disk.nvme1n1.in‚201379.3828125# disk.nvme1n1.out‚61077.19921875* proc.memory.availableMB‚10120.42578125 disk.sda.in‚5.5625# disk.nvme0n1.in‚329959.16015625! disk.sda2.out‚1017.3173828125 disk.md0p1.out‚ 980.5703125! disk.sdb2.out‚1017.3173828125 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 2.59619140625 disk.sdb1.out‚0 disk.sda1.in‚0 network.sent‚ 2.072573e+07) disk./.usagePercent‚83.16371510664689 disk.sda.out‚1017.3173828125 disk.md0.out‚ 980.5703125 disk.sdb.in‚5.40625# disk.nvme0n1.out‚54948.22265625 disk.dm-0.out‚ 116025.421875" proc.memory.rssMB‚ 3032.2265625 proc.cpu.threads‚17% disk./.usageGB‚2974.3372764587402 disk.sda1.out‚0 disk.md0.in‚10.96875 disk.sdb1.in‚0 disk.dm-0.in‚531338.54296875 disk.md0p1.in‚10.96875 disk.rbd2.out‚ 2.74609375æäã2ç"- _wandbruntime‚9277  _runtime‚9277› <ÖÇè"Á _step‚650 & lr-AdamW/pg1‚0.0002611642669713081 & lr-AdamW/pg2‚0.0002611642669713081 trainer/global_step‚7949 % _timestamp‚1.7884732247732465e+09 _runtime‚9277.958199097þ6t Ìé"Æ trainer/global_step‚7949 % _timestamp‚1.7884732247732465e+09 _runtime‚9277.958199097 _step‚650 & lr-AdamW/pg1‚0.0002611642669713081 & lr-AdamW/pg2‚0.0002611642669713081Šd¯ñ2ê"- _wandbruntime‚9277  _runtime‚9277*8]øâë"åO JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚323791153075398 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚323706570758144 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90300416 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 147666586 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚339501250469376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15794679711232 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚679 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚308041478023536 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚339500998797824 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 301105152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151113137 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 367240511 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚7949 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 216127374 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚358942869049856 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16061464846336 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚339500998797824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57366851 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151113137 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151113135 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚323706570758144 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15794446042624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚342881404203520 " _timestamp‚1.7884732247743e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15749675051862 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 _step‚651 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 216101030 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90300414 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15794446042624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚681 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15794679711232 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325045248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273°=çð'ì"êO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚323706570758144 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15794446042624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚342881404203520 " _timestamp‚1.7884732247743e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15749675051862 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 _step‚651 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 216101030 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90300414 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15794446042624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚681 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15794679711232 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325045248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚323791153075398 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚323706570758144 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90300416 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 147666586 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚339501250469376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15794679711232 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚679 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚308041478023536 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚339500998797824 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 301105152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151113137 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 367240511 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚7949 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 216127374 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚358942869049856 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16061464846336 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚339500998797824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57366851 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151113137 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151113135 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0‹/¼å2í"- _wandbruntime‚9278  _runtime‚9278YFlçø&î"òM F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 trainer/global_step‚7949 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 216154092 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15796410068480 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 216127722 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚358986862896640 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 151131410 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚339542248860672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90311218 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15796643750400 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16063460176896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚308079008380960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 216154092 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚323745856795136 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 367259132 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚323830887942460 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151131410 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151131412 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16063750782976 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚323745838792192 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15796643750400 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 367285504 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151131412 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚358986548350464 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚339542500545536 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 147685351 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚323745856795136 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 367259132 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15751862522140 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15751633540848 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57374133 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚323830641921808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884732257336547e+09 _step‚652 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147684679k3逺%ï"÷M B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15751862522140 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15751633540848 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57374133 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚323830641921808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884732257336547e+09 _step‚652 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147684679 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 trainer/global_step‚7949 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 216154092 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15796410068480 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 216127722 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚358986862896640 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 151131410 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚339542248860672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90311218 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15796643750400 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16063460176896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚308079008380960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 216154092 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚323745856795136 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 367259132 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚323830887942460 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151131410 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151131412 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16063750782976 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚323745838792192 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15796643750400 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 367285504 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151131412 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚358986548350464 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚339542500545536 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 147685351 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚323745856795136 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 367259132 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚341563516354048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚309918012934096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚341563768025600 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚341563768025600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16161316720640 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚344965626646016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚361127231888896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57715001 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15892895417856 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90840516 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15847613092370 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 217438982 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57714314 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _€ ó'€#íO UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 152026887 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚361126919426560 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚344965602705920 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚325670872607744 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 369465867 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚325670854604800 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90840514 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚309917995894736 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90840516 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15847613092370 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 217438982 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57714314 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 148554828 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152026885 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 217465326 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 148555517 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15892661749248 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 369492213 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚341563516354048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 217465326 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚325670872607744 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 _step‚655 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15892895417856 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16161605242880 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚341563516354048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚309918012934096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚341563768025600 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚341563768025600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16161316720640 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚344965626646016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚361127231888896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57715001 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15892895417856 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 369537206 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152045162 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 217465674 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 369537206 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16163309233152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚687 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57721807 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚325710158644736 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚325710140641792 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚325805097833516 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15849800562648 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚341605018101760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 148573125 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚325805343854168 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148572438 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 152045160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16163599839232 _step‚656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57721122 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 217492044 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15894625775104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90851316 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚345007286675968 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 369510834 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 trainer/global_step‚7999 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 369510834 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚345007310616064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚309955543291520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚361170910455296 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚341604766416896 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚361170595909120 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104¯¤7˜ý&ƒ#÷M UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚325710158644736 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚325710140641792 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚325805097833516 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15849800562648 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚341605018101760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 148573125 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚325805343854168 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148572438 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 152045160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16163599839232 _step‚656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57721122 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 217492044 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15894625775104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90851316 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚345007286675968 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 369510834 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 trainer/global_step‚7999 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 369510834 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚345007310616064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚309955543291520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚361170910455296 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚341604766416896 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚361170595909120 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 152045162 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 152045160 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15894625775104 _runtime‚9332.139894259 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 217492044 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚341605018101760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 369537206 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152045162 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 217465674 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 369537206 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16163309233152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚687 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57721807 FR_†#Ù /loss/FRt1/assignment_loss‚3.191840846836567 /loss/FRt4/detection_loss‚3.2564141154289246 1loss/SRqqt2/detection_loss‚2.2586617435727803 .loss/FBt4/detection_loss‚2.527506397529082 .loss/FBt1/detection_loss‚2.458587035536766 /loss/FRt2/detection_loss‚2.1367471182399407 /loss/FRt3/detection_loss‚1.2280494443379169 _step‚657 1loss/FBt2/assignment_loss‚0.19955555369167777 epoch‚6 /loss/FRt4/assignment_loss‚2.420643796091494 1loss/SRqqt3/detection_loss‚0.7356977972819984 trainer/global_step‚7999 _runtime‚9332.140957644 0loss/FRt2/assignment_loss‚3.7340927844823795 2loss/SRqqt2/assignment_loss‚1.6569694609514305 2loss/SRqqt3/assignment_loss‚0.8267284889583624 1loss/FBt3/assignment_loss‚0.08631096511231663 0loss/FBt4/assignment_loss‚0.2439210899171875 1loss/SRqqt1/detection_loss‚2.2474695121248565 /loss/FBt3/detection_loss‚0.4794263907431348 0loss/FRt3/assignment_loss‚0.8785964798095615 1loss/SRqqt4/assignment_loss‚1.960236353580566 0loss/FBt1/assignment_loss‚0.2903793913865229 /loss/FRt1/detection_loss‚2.6558117300271986 %loss/total_loss‚41.33119373413413 1loss/SRqqt1/assignment_loss‚1.553477219823334 1loss/SRqqt4/detection_loss‚2.1605752846773933 .loss/FBt2/detection_loss‚2.143494733992745 % _timestamp‚1.7884732789562361e+09‘}è…=‘‡#‚(:† ËßçÔäÑ؆ gpu.0.gpu‚11.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚145.438' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.3595 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0UpS~ˆ#‚(:ó ËßçÔЀö†) proc.memory.availableMB‚ 10120.0703125 network.recv‚ 3.196094e+06) disk./.usagePercent‚83.15692467052989 disk.rbd2.in‚0 disk.sda2.in‚5.5625 disk.sdb1.out‚0# disk.nvme0n1.out‚55027.40234375 disk.dm-0.in‚531338.54296875! disk.sdb.out‚1025.55810546875 network.sent‚ 2.0856154e+07% disk./.usageGB‚2974.0944175720215" disk.nvme1n1.in‚201379.3828125 disk.sdb2.in‚5.40625 disk.rbd0.in‚0.015625 disk.sda.in‚5.5625# disk.nvme0n1.in‚329959.16015625 disk.sda1.out‚0" disk.sdb2.out‚1025.55810546875$ memory_percent‚38.23199272155762" proc.memory.rssMB‚ 3032.2265625) proc.memory.percent‚18.50724220275879 proc.cpu.threads‚17 disk.md0p1.in‚10.96875 cpu‚25.186147031557343# disk.nvme1n1.out‚61477.54296875! disk.sda.out‚1025.55810546875 disk.sdb.in‚5.40625 disk.md0.in‚10.96875 disk.rbd0.out‚ 2.59619140625 disk.rbd2.out‚ 2.74609375 disk.dm-0.out‚116504.9453125" disk.sda2.out‚1025.55810546875 disk.md0.out‚ 988.48828125 disk.sdb1.in‚0 disk.sda1.in‚0 disk.md0p1.out‚ 988.48828125·€d‰#‚(:… ÚßçÔÃú‡ gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚142.93' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.7325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…8ÅHxŠ#‚(:í ÚßçÔÂ͡ network.recv‚ 3.203716e+06# disk.nvme0n1.in‚329959.16015625 disk.dm-0.in‚531338.54296875 disk.rbd0.out‚ 2.59619140625 disk.sdb.in‚5.40625$ memory_percent‚38.23196887969971 proc.cpu.threads‚17 disk.sda1.in‚0 disk.md0.out‚ 990.046875" disk.nvme1n1.out‚ 61557.2734375 disk.sdb1.in‚0" disk.sdb2.out‚1027.18212890625 disk.rbd0.in‚0.015625 disk.sdb1.out‚0* proc.memory.availableMB‚10120.07421875" disk.nvme1n1.in‚201379.3828125 network.sent‚ 2.0964151e+07! disk.dm-0.out‚116657.13671875 disk.sda1.out‚0 disk.md0p1.out‚ 990.046875" proc.memory.rssMB‚ 3032.2265625) disk./.usagePercent‚83.16039988398747# disk./.usageGB‚2974.21870803833 disk.sda.in‚5.5625# disk.nvme0n1.out‚55099.86328125 disk.sdb2.in‚5.40625 disk.rbd2.out‚ 2.74609375) proc.memory.percent‚18.50724220275879 disk.md0p1.in‚10.96875 disk.md0.in‚10.96875 disk.rbd2.in‚0 cpu‚25.18587269827914! disk.sda.out‚1027.18212890625 disk.sda2.in‚5.5625" disk.sda2.out‚1027.18212890625! disk.sdb.out‚1027.18212890625Ñúzב‹#‚(:† éßçÔúµ›† gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚134.298' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.5745 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉX™(VŒ#‚(:Ë éßçÔëÊʇ disk.md0p1.in‚10.96875 disk.md0p1.out‚ 992.078125 disk.rbd0.in‚0.015625! disk.nvme1n1.out‚ 61653.796875 disk.sdb.out‚ 1029.28125 disk.dm-0.in‚531339.4140625 network.recv‚ 3.208181e+06# disk.nvme0n1.out‚55172.33984375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚5.5625 disk.sda.in‚5.5625 disk.sda.out‚ 1029.28125 proc.cpu.threads‚17 network.sent‚ 2.0969975e+07 disk.rbd2.out‚ 2.74609375) proc.memory.percent‚18.50724220275879 disk.sdb2.in‚5.40625 disk.sdb2.out‚ 1029.28125 disk.sdb.in‚5.40625 disk.rbd0.out‚ 2.59619140625 disk.sdb1.out‚0! disk.dm-0.out‚116826.13671875 disk.sda2.out‚ 1029.28125 disk.md0.in‚10.96875 disk.md0.out‚ 992.078125 disk.nvme1n1.in‚201380 disk.sdb1.in‚0) proc.memory.availableMB‚ 10119.8359375" proc.memory.rssMB‚ 3032.2265625 cpu‚25.185576383657423) disk./.usagePercent‚83.17091511920722$ disk./.usageGB‚2974.594783782959 disk.rbd2.in‚0" disk.nvme0n1.in‚329959.4140625$ memory_percent‚38.23342323303223yr±Ê2#- _wandbruntime‚9385  _runtime‚9385b®ÇŽ#Á trainer/global_step‚8049 % _timestamp‚1.7884733319483986e+09 _runtime‚9385.133407649 _step‚658 & lr-AdamW/pg1‚0.0002501547940279748 & lr-AdamW/pg2‚0.0002501547940279748ˆ‘hŠÌ#Æ trainer/global_step‚8049 % _timestamp‚1.7884733319483986e+09 _runtime‚9385.133407649 _step‚658 & lr-AdamW/pg1‚0.0002501547940279748 & lr-AdamW/pg2‚0.0002501547940279748’†ZB,2#- _wandbruntime‚9385  _runtime‚9385 «Ý¡Ò‘#çO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚681 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58060706 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚343626285581824 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 218776934 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 149440641 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 trainer/global_step‚8049 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚327635174457344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15945780113962 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 371717569 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚343626033910272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15945551132878 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 152940637 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15990877455872 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15991111124480 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚327635156454400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 218776934 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15991111124480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚363310988685824 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚343626033910272 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 149441322 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚343626285581824 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚327740064898814 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58060027 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚311794513765936 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152940635 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91380614 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚363311294856704 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306170880 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16261469708288 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 _step‚659 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚327635174457344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 218803278 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚679 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 $ _timestamp‚1.788473331949564e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 371743915 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582êH °ò'’#ìO [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15990877455872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91380616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58060027 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚311794513765936 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152940635 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91380614 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚363311294856704 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306170880 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16261469708288 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 _step‚659 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚327635174457344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 218803278 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚679 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 $ _timestamp‚1.788473331949564e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 371743915 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚681 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58060706 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚343626285581824 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 218776934 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 149440641 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 trainer/global_step‚8049 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚327635174457344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15945780113962 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 371717569 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚343626033910272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15945551132878 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 152940637 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15990877455872 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15991111124480 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚327635156454400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 218776934 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15991111124480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚363310988685824 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚343626033910272 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 149441322 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚343626285581824 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚327740064898814 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91391416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58067787 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15992841481728 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 371788908 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚698 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚363354978043392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 152958910 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚327779799765876 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 218829996 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚327779553745224 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 152958912 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚347091509118464 _step‚660 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚311832061162720 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 371762536 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16263468924928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚343667283973120 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚363354663497216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚700 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚327674442491392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 371762536 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 218829996 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 218803626 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15993075163648 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 371788908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚343667535657984 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚327674460494336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚343667535657984 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152958912 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 152958910 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 149458505 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58067089 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15947509621864 _runtime‚9386.106599663 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚311832044123360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 152958910 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚327779799765876 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 218829996 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚327779553745224 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 152958912 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚347091509118464 _step‚660 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚311832061162720 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 371762536 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16263468924928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚343667283973120 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚363354663497216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚700 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚327674442491392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 371762536 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 218829996 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 218803626 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15993075163648 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 371788908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚343667535657984 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚327674460494336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚343667535657984 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152958912 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 152958910 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 149458505 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58067089 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15947509621864 _runtime‚9386.106599663 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚311832044123360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91391416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58067787 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15992841481728 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 371788908 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚698 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚363354978043392 TFDeviceStatsMonitor.on_train_batch_end/ina.Õ·öWctive_split_bytes.all.current‚ 314546176 P¾dœ#‚(:Ù ‡àçÔÚšóˆ disk.sdb1.out‚0 disk.md0p1.out‚994.8125 disk.sdb2.in‚5.40625$ memory_percent‚38.23087215423584) disk./.usagePercent‚83.17787919867051 disk.sdb1.in‚0 disk.md0.out‚994.8125 disk.sdb.out‚1032.134765625! disk.dm-0.out‚117177.74609375 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0.in‚10.96875 disk.rbd2.out‚ 2.74609375* proc.memory.availableMB‚10120.25390625 cpu‚25.185032313608236 disk.dm-0.in‚531339.4140625 disk.sdb2.out‚1032.134765625# disk.nvme0n1.out‚55316.66015625 disk.sda2.out‚1032.134765625 disk.md0p1.in‚10.96875 disk.rbd2.in‚0 disk.sda.out‚1032.134765625 network.recv‚ 3.217974e+06" disk.nvme1n1.out‚ 61861.0859375 disk.sdb.in‚5.40625) proc.memory.percent‚18.50724220275879 proc.cpu.threads‚17 disk.sda.in‚5.5625" disk.nvme0n1.in‚329959.4140625 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 2.65869140625" proc.memory.rssMB‚ 3032.2265625 network.sent‚ 2.1084088e+07$ disk./.usageGB‚2974.843852996826 disk.nvme1n1.in‚201380 disk.sda2.in‚5.5625µƒ¸’#‚(:‡ –àçÔŒ³ç‡ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚139.251' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.81275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏÍÊ’zž#‚(:ï –àçÔ…ñ͈ disk.rbd0.out‚ 2.65869140625 disk.rbd2.out‚ 2.74609375" disk.nvme0n1.in‚329959.4140625 disk.sda2.in‚ 5.56640625$ memory_percent‚38.23540210723877 disk.sda.out‚1033.7548828125! disk.dm-0.out‚117360.40234375) proc.memory.percent‚18.50724220275879 network.recv‚ 3.22238e+06$ disk./.usageGB‚2974.843795776367 disk.rbd0.in‚0.015625 disk.dm-0.in‚531339.4140625 disk.sda1.in‚0 cpu‚25.184740142809183 network.sent‚ 2.1088229e+07) disk./.usagePercent‚83.17787759876254 disk.md0.out‚ 996.37109375 disk.sdb.in‚5.40625! disk.sda2.out‚1033.7548828125 disk.md0p1.out‚ 996.37109375 disk.rbd2.in‚0 disk.sdb.out‚1033.7548828125# disk.nvme0n1.out‚55388.71484375" proc.memory.rssMB‚ 3032.2265625 proc.cpu.threads‚17 disk.sdb2.in‚5.40625 disk.nvme1n1.out‚ 61971.6875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0p1.in‚ 10.97265625* proc.memory.availableMB‚10119.51171875! disk.sdb2.out‚1033.7548828125 disk.md0.in‚ 10.97265625 disk.nvme1n1.in‚201380 disk.sda.in‚ 5.56640625 disk.sda1.out‚0/=gE’Ÿ#‚(:‡ ¥àçÔ§Œš† gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚144.489' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.12225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üTB v #‚(:ë ¥àçÔÓù˜‡) proc.memory.percent‚18.50724220275879 network.recv‚ 3.226857e+06% disk./.usageGB‚2975.3458137512207! disk.sda2.out‚1036.0166015625 disk.sdb.out‚1036.0166015625 disk.sda1.out‚0 disk.md0p1.out‚ 998.57421875 disk.sdb2.in‚5.40625! disk.sdb2.out‚1036.0166015625 disk.nvme1n1.in‚201380 network.sent‚ 2.1094272e+07" disk.nvme0n1.out‚ 55460.8984375 disk.sda1.in‚0 disk.md0p1.in‚ 10.97265625 disk.md0.in‚ 10.97265625 disk.sdb.in‚5.40625" proc.memory.rssMB‚ 3032.2265625 cpu‚25.1844817695524 disk.sda2.in‚ 5.56640625 disk.md0.out‚ 998.57421875 disk.rbd0.out‚ 2.6826171875 disk.rbd2.in‚0 disk.sdb1.in‚0 disk.sda.out‚1036.0166015625" disk.nvme0n1.in‚329959.4140625 disk.rbd0.in‚0.015625 disk.sdb1.out‚0 proc.cpu.threads‚17 disk.sda.in‚ 5.56640625 disk.rbd2.out‚ 2.74609375 disk.nvme1n1.out‚ 62049.28125$ memory_percent‚38.23537826538086) disk./.usagePercent‚83.19191423145028 disk.dm-0.in‚531339.4140625 disk.dm-0.out‚117510.1796875( proc.memory.availableMB‚ 10119.515625ÇYcq2¡#- _wandbruntime‚9440  _runtime‚9440zp`ÊÇ¢#Á & lr-AdamW/pg1‚0.0002447001488713461 & lr-AdamW/pg2‚0.0002447001488713461 trainer/global_step‚8099 % _timestamp‚1.7884733872065609e+09 _runtime‚9440.391558752 _step‚662‘ŽP”Ì£#Æ & lr-AdamW/pg1‚0.0002447001488713461 & lr-AdamW/pg2‚0.0002447001488713461 trainer/global_step‚8099 % _timestamp‚1.7884733872065609e+09 _runtime‚9440.391558752 _step‚662–[ÀÈ®2¤#- _wandbruntime‚9440  _runtime‚9440-·ì1î'¥#èO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91920714 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58409913 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚345688551466496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 220114886 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 220141230 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16089326831104 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚365495146311168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚329714766830966 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16089093162496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153854385 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 150330627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58410609 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚349133999710720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚8099 % _timestamp‚1.7884733872073529e+09 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16361146600448 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 153854385 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304073728 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 220114886 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚345688803138048 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 150331325 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚698 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚349134023650816 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 373969271 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16089093162496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 373995617 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 220141230 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16361426734080 _step‚663 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91920716 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚345688803138048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚329599458304000 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚313671048676496 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153854387 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280133632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚345688551466496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚329599476306944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚329599458304000 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16089326831104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16043718154470 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 _runtime‚9440.392748477 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 153854387 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582÷—›gó'¦#íO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚345688803138048 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 150331325 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚698 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚349134023650816 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 373969271 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16089093162496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 373995617 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 220141230 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16361426734080 _step‚663 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91920716 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚345688803138048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚329599458304000 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚313671048676496 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153854387 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280133632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚345688551466496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚329599476306944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚329599458304000 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16089326831104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16043718154470 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 _runtime‚9440.392748477 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 153854387 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91920714 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58409913 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚345688551466496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 220114886 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 220141230 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16089326831104 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚365495146311168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚329714766830966 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16089093162496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153854385 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 150330627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58410609 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚349133999710720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚8099 % _timestamp‚1.7884733872073529e+09 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16361146600448 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 153854385 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304073728 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 220114886—ÿ¼1ö2§#- _wandbruntime‚9441  _runtime‚9441Xó+¨#òM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 220141578 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚329638762343936 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚329754255677584 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚9441.467042758 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚329638744340992 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16091290870272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 220141578 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 FDeviceStatsMonitor.on_trai_V›IÍn_batch_end/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91931516 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58417155 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16363428455424 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚313708579033920 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91931518 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚329754009656932 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚345730053214208 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16091057188352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _step‚664 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 220167948 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 150348673 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16091057188352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚329638744340992 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16091290870272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 220167948 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16045676643664 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 374040610 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚345730053214208 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153872660 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150347992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 trainer/global_step‚8099 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16045447662372 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 153872662 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58416476 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚313708561994560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚681 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚349175683680768 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 374014238 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 374014238 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153872662 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚329638762343936 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚365539136076288 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚345729801529344 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16363137849344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884733882814848e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚365538821530112 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 374040610 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚345729801529344 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16363137849344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884733882814848e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚365538821530112 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 374040610 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚9441.467042758 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚329638744340992 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16091290870272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 220141578 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91931516 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58417155 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16363428455424 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚313708579033920 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91931518 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚329754009656932 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚345730053214208 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16091057188352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _step‚664 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 220167948 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 150348673 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16091057188352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚329638744340992 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16091290870272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 220167948 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16045676643664 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 374040610 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚345730053214208 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153872660 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150347992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 trainer/global_step‚8099 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16045447662372 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 153872662 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58416476 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚313708561994560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚681 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚349175683680768 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 374014238 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 374014238 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360˜}H}ˆ2ª#- _wandbruntime‚9441  _runtime‚9441´Â‡Û_«#Ù 0loss/FRt3/assignment_loss‚1.2668240714457728 2loss/SRqqt3/assignment_loss‚0.9460611115574983 _runtime‚9441.468182313 /loss/FRt1/assignment_loss‚2.847595208569577 0loss/FRt4/assignment_loss‚2.6637158592542014 2loss/SRqqt1/assignment_loss‚1.3753240717136686 1loss/FBt1/assignment_loss‚0.09349750981899971 .loss/FRt4/detection_loss‚2.895865731769138 1loss/SRqqt1/detection_loss‚2.1444355214343354 /loss/FBt2/detection_loss‚2.0067476704716682 1loss/FBt3/assignment_loss‚0.10982049894612043 /loss/FRt1/detection_loss‚2.6091557270602177 /loss/FRt3/detection_loss‚1.6127833474667803 0loss/FBt2/assignment_loss‚0.2740243954710081 1loss/SRqqt3/detection_loss‚0.9697827651147165 0loss/SRqqt4/detection_loss‚2.249935647249222 0loss/FBt3/detection_loss‚0.42381437050343884 %loss/total_loss‚39.96509390397863 epoch‚6 _step‚665 2loss/SRqqt4/assignment_loss‚1.5730879801511763 1loss/SRqqt2/detection_loss‚2.5487147064312645 .loss/FBt1/detection_loss‚2.800596470182592 /loss/FBt4/detection_loss‚2.1603420128424964 % _timestamp‚1.7884733882834594e+09 2loss/SRqqt2/assignment_loss‚1.1906834084378637 1loss/FBt4/assignment_loss‚0.23441808923637097 /loss/FRt2/detection_loss‚1.9731946661550965 0loss/FRt2/assignment_loss‚2.9946730626953975 trainer/global_step‚8099÷v¥dd¬#Þ 1loss/FBt3/assignment_loss‚0.10982049894612043 /loss/FRt1/detection_loss‚2.6091557270602177 /loss/FRt3/detection_loss‚1.6127833474667803 0loss/FBt2/assignment_loss‚0.2740243954710081 1loss/SRqqt3/detection_loss‚0.9697827651147165 0loss/SRqqt4/detection_loss‚2.249935647249222 0loss/FBt3/detection_loss‚0.42381437050343884 %loss/total_loss‚39.96509390397863 epoch‚6 _step‚665 2loss/SRqqt4/assignment_loss‚1.5730879801511763 1loss/SRqqt2/detection_loss‚2.5487147064312645 .loss/FBt1/detection_loss‚2.800596470182592 /loss/FBt4/detection_loss‚2.1603420128424964 % _timestamp‚1.7884733882834594e+09 2loss/SRqqt2/assignment_loss‚1.1906834084378637 1loss/FBt4/assignment_loss‚0.23441808923637097 /loss/FRt2/detection_loss‚1.9731946661550965 0loss/FRt2/assignment_loss‚2.9946730626953975 trainer/global_step‚8099 0loss/FRt3/assignment_loss‚1.2668240714457728 2loss/SRqqt3/assignment_loss‚0.9460611115574983 _runtime‚9441.468182313 /loss/FRt1/assignment_loss‚2.847595208569577 0loss/FRt4/assignment_loss‚2.6637158592542014 2loss/SRqqt1/assignment_loss‚1.3753240717136686 1loss/FBt1/assignment_loss‚0.09349750981899971 .loss/FRt4/detection_loss‚2.895865731769138 1loss/SRqqt1/detection_loss‚2.1444355214343354 /loss/FBt2/detection_loss‚2.0067476704716682™¯tUÀ‘­#‚(:† ´àçÔí版 gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.966' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.9915 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çÏ{ˆ®#‚(:ý ´àçÔÙ›‡Š disk.md0p1.in‚ 10.97265625 disk.sdb2.in‚5.40625 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.rbd2.out‚ 2.74609375 cpu‚25.184200492457407 disk.md0p1.out‚ 999.88671875 disk.md0.in‚ 10.97265625) proc.memory.percent‚18.50724220275879 disk.dm-0.in‚531339.43359375 disk.md0.out‚ 999.88671875 disk.rbd0.out‚ 2.6826171875 disk.sdb.out‚1037.3876953125 network.sent‚ 2.1113077e+07% disk./.usageGB‚2975.3439025878906! disk.sda2.out‚1037.3876953125 network.recv‚ 3.230869e+06 disk.sda1.out‚0 disk.sda.in‚ 5.56640625" proc.memory.rssMB‚ 3032.2265625 disk.sda1.in‚0! disk.sdb2.out‚1037.3876953125# disk.nvme1n1.in‚201380.01953125 disk.sdb.in‚5.40625 disk.sdb1.in‚0 disk.sdb1.out‚0" disk.nvme0n1.in‚329959.4140625# disk.nvme0n1.out‚55532.96484375 disk.dm-0.out‚117685.6796875 disk.sda2.in‚ 5.56640625 disk.sda.out‚1037.3876953125* proc.memory.availableMB‚10119.98046875 proc.cpu.threads‚17# disk.nvme1n1.out‚62152.71484375$ memory_percent‚38.23254108428955) disk./.usagePercent‚83.19186079452369èU¯#‚(:… ÃàçÔîÔ‡ gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚135.184' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.796 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÍÛU튰#‚(:ÿ Ãàçԣň disk.sdb.in‚5.40625 disk.md0.out‚ 1001.40234375 disk.rbd2.out‚ 2.74609375# disk.nvme1n1.in‚201380.01953125 disk.sda1.in‚0! disk.sda2.out‚1038.9638671875 disk.md0p1.out‚ 1001.40234375! disk.sdb2.out‚1038.9638671875 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 2.74560546875 disk.dm-0.out‚117887.1328125 network.sent‚ 2.1209409e+07 disk.dm-0.in‚531339.43359375 disk.sdb1.in‚0" proc.memory.rssMB‚ 3032.2265625 disk.sda1.out‚0 disk.sdb2.in‚5.40625$ memory_percent‚38.23254108428955% disk./.usageGB‚2975.3438453674316" disk.nvme0n1.in‚329959.4140625 disk.sda2.in‚ 5.56640625 disk.md0.in‚ 10.97265625 disk.rbd2.in‚0* proc.memory.availableMB‚10119.98046875) proc.memory.percent‚18.50724220275879( disk./.usagePercent‚83.1918591946157 disk.sdb1.out‚0 disk.sda.in‚ 5.56640625 disk.sda.out‚1038.9638671875# disk.nvme0n1.out‚55605.10546875# disk.nvme1n1.out‚62282.02734375 proc.cpu.threads‚17 network.recv‚ 3.237656e+06 disk.sdb.out‚1038.9638671875 disk.md0p1.in‚ 10.97265625 cpu‚25.183970595741386àª)v’±#‚(:‡ ÒàçÔžñ¯† gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚134.575' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.64375 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•†²#‚(:‚  ÒàçÔƒ’ú‡ proc.cpu.threads‚17# disk.nvme1n1.in‚201380.01953125" disk.nvme0n1.in‚329959.4140625# disk.nvme0n1.out‚55677.16015625 disk.sda1.in‚0" proc.memory.rssMB‚ 3032.2265625) proc.memory.percent‚18.50724220275879 cpu‚25.183683834172157( disk./.usagePercent‚83.1918591946157 disk.sda.in‚ 5.56640625! disk.sda.out‚1040.86181640625 network.recv‚ 3.242271e+06 disk.rbd0.out‚ 2.74560546875 disk.rbd2.in‚0" disk.nvme1n1.out‚ 62282.1015625) proc.memory.availableMB‚ 10119.7265625 disk.md0p1.in‚ 10.97265625 disk.md0p1.out‚ 1003.23046875 disk.sdb.in‚5.40625! disk.dm-0.out‚117959.26171875" disk.sda2.out‚1040.86181640625" disk.sdb2.out‚1040.86181640625 disk.md0.in‚ 10.97265625 disk.rbd2.out‚ 2.74609375 disk.sdb1.out‚0 disk.sda2.in‚ 5.56640625 disk.md0.out‚ 1003.23046875 disk.rbd0.in‚0.015625! disk.sdb.out‚1040.86181640625 network.sent‚ 2.1213526e+07 disk.sda1.out‚0 disk.sdb2.in‚5.40625 disk.sdb1.in‚0% disk./.usageGB‚2975.3438453674316 disk.dm-0.in‚531339.43359375$ memory_percent‚38.23409080505371‚Ü,œ³#‚(:‘ áàçÔíï† gpu.0.gpu‚10.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0d’·Z…´#‚(:ú áàçÔ¡ýƇ disk.sdb.in‚5.40625 disk.sda.in‚ 5.56640625! disk.sda.out‚1042.39501953125 disk.sda1.in‚0" disk.sda2.out‚1042.39501953125 disk.md0p1.in‚ 10.97265625 disk.sda2.in‚ 5.56640625% disk./.usageGB‚2975.3439025878906 disk.md0.out‚ 1004.6953125 disk.rbd2.out‚ 2.74609375# disk.nvme1n1.in‚201380.01953125 disk.sdb1.out‚0) proc.memory.availableMB‚ 10119.9765625) proc.memory.percent‚18.50724220275879 disk.md0.in‚ 10.97265625! disk.sdb.out‚1042.39501953125" disk.nvme0n1.in‚329959.5078125 disk.dm-0.in‚531339.52734375" proc.memory.rssMB‚ 3032.2265625 cpu‚25.183396862837757" disk.sdb2.out‚1042.39501953125 disk.md0p1.out‚ 1004.6953125$ memory_percent‚38.23256492614746 network.recv‚ 3.246634e+06 disk.sdb2.in‚5.40625 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 2.80859375 disk.nvme0n1.out‚ 55749.34375 disk.dm-0.out‚118212.4140625 proc.cpu.threads‚17 disk.sda1.out‚0) disk./.usagePercent‚83.19186079452369" disk.nvme1n1.out‚ 62463.0703125 disk.sdb1.in‚0 network.sent‚ 2.1219515e+07 disk.rbd2.in‚0ðpÛ2µ#- _wandbruntime‚9497  _runtime‚9497þ5¬É¶#à ' lr-AdamW/pg2‚0.00023928041592416178 trainer/global_step‚8149 % _timestamp‚1.7884734439671714e+09 _runtime‚9497.152182223 _step‚666 ' lr-AdamW/pg1‚0.00023928041592416178’†Sη#È _runtime‚9497.152182223 _step‚666 ' lr-AdamW/pg1‚0.00023928041592416178 ' lr-AdamW/pg2‚0.00023928041592416178 trainer/global_step‚8149 % _timestamp‚1.7884734439671714e+09š™<¥2¸#- _wandbruntime‚9497  _runtime‚9497|fïU¹#çO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16141656194978 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16141427213894 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 221479182 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚315547566547696 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16187542537728 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16187542537728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚347751069022720 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884734439683855e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 376247319 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚331563760153600 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚351218222153216 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 154768137 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚347751320694272 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚698 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚347751069022720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16461252894720 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 151218332 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16187308869120 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚367679162585600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚8149 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚367679475047936 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚331688976722230 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _runtime‚ 9497.15354248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 221452838 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154768137 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚92460816 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚92460814 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 154768135 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16187308869120 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 151217632 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58757516 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712S8¶Ôò'º#ìO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚331688976722230 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _runtime‚ 9497.15354248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 221452838 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154768137 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚92460816 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚92460814 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 154768135 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16187308869120 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 151217632 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58757516 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16141656194978 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16141427213894 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 221479182 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚315547566547696 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16187542537728 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16187542537728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚347751069022720 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884734439683855e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 376247319 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚331563760153600 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚351218222153216 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 154768137 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚347751320694272 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚698 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚347751069022720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16461252894720 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 151218332 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16187308869120 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚367679162585600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚8149 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚367679475047936›]…2»#- _wandbruntime‚9498  _runtime‚9498ý•ö1ø&¼#òM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58763563 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154786412 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 376292312 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚347792319085568 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16189506576896 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154786412 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16463250280448 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 trainer/global_step‚8149 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16189506576896 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 221479530 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚700 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚331603064193536 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚347792570770432 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151235179 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 151235881 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚331603046190592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚331728465568640 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚315585096905120 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16143385702880 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 154786410 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚331603046190592 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚347792570770432 _step‚668 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚351259906123264 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 154786410 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92471616 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 376265940 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚331603064193536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚92471618 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 221505900 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚9498.233758974 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 376292312 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16462959674368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚331728711589292 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 376265940 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚367722841857536 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 221505900 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16189272894976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚702 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚351259882183168 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 221479530 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16143614684172 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0Sc5)½#÷M I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚331603046190592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚331728465568640 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚315585096905120 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16143385702880 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 154786410 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚331603046190592 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚347792570770432 _step‚668 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚351259906123264 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 154786410 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92471616 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 376265940 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚331603064193536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚92471618 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 221505900 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚9498.233758974 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 376292312 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16462959674368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚331728711589292 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 376265940 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚367722841857536 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 221505900 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16189272894976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚702 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚351259882183168 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 221479530 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16143614684172 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58763563 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154786412 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 376292312 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H@DeviceStatsMonitor.on_trA#ÏÔ ain_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚347792319085568 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16189506576896 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154786412 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16463250280448 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 trainer/global_step‚8149 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16189506576896 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 221479530 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚700 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚331603064193536 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚347792570770432 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151235179 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 151235881 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104œÁ‡.Ü2¾#- _wandbruntime‚9498  _runtime‚9498,¬KZ¿#Ô 1loss/FBt4/assignment_loss‚0.39436152049641277 1loss/SRqqt3/detection_loss‚0.7917662034350856 _step‚669 /loss/FRt4/assignment_loss‚3.027552428068938 .loss/FRt1/detection_loss‚2.664929791804283 /loss/FRt4/detection_loss‚2.9096923028981245 /loss/FBt1/detection_loss‚2.3282092064619064 /loss/FRt3/detection_loss‚1.5570283266424667 0loss/SRqqt1/detection_loss‚1.832874412669076 _runtime‚9498.235184021 0loss/FRt3/assignment_loss‚1.1438940400257707 1loss/FBt3/assignment_loss‚0.05187684163935726 0loss/SRqqt2/detection_loss‚2.757465465068817 .loss/FBt2/detection_loss‚1.753348964711894 /loss/FRt2/detection_loss‚2.1247683366005403 1loss/SRqqt4/detection_loss‚2.3764569044113157 epoch‚6 0loss/FRt2/assignment_loss‚3.4026378301473765 0loss/FBt1/assignment_loss‚0.0965006576452538 0loss/FBt3/detection_loss‚0.48122102329981026 $ _timestamp‚1.788473445050458e+09 .loss/FBt4/detection_loss‚2.576450392603874 1loss/SRqqt2/assignment_loss‚2.203061020374298 2loss/SRqqt3/assignment_loss‚0.9445826857885992 1loss/FBt2/assignment_loss‚0.20726788170544994 %loss/total_loss‚41.11425498937318 trainer/global_step‚8149 0loss/FRt1/assignment_loss‚3.0195218324661255 2loss/SRqqt1/assignment_loss‚1.1202976102630298 1loss/SRqqt4/assignment_loss‚1.348489310145378-lK±_À#Ù .loss/FBt4/detection_loss‚2.576450392603874 1loss/SRqqt2/assignment_loss‚2.203061020374298 2loss/SRqqt3/assignment_loss‚0.9445826857885992 1loss/FBt2/assignment_loss‚0.20726788170544994 %loss/total_loss‚41.11425498937318 trainer/global_step‚8149 0loss/FRt1/assignment_loss‚3.0195218324661255 2loss/SRqqt1/assignment_loss‚1.1202976102630298 1loss/SRqqt4/assignment_loss‚1.348489310145378 1loss/FBt4/assignment_loss‚0.39436152049641277 1loss/SRqqt3/detection_loss‚0.7917662034350856 _step‚669 /loss/FRt4/assignment_loss‚3.027552428068938 .loss/FRt1/detection_loss‚2.664929791804283 /loss/FRt4/detection_loss‚2.9096923028981245 /loss/FBt1/detection_loss‚2.3282092064619064 /loss/FRt3/detection_loss‚1.5570283266424667 0loss/SRqqt1/detection_loss‚1.832874412669076 _runtime‚9498.235184021 0loss/FRt3/assignment_loss‚1.1438940400257707 1loss/FBt3/assignment_loss‚0.05187684163935726 0loss/SRqqt2/detection_loss‚2.757465465068817 .loss/FBt2/detection_loss‚1.753348964711894 /loss/FRt2/detection_loss‚2.1247683366005403 1loss/SRqqt4/detection_loss‚2.3764569044113157 epoch‚6 0loss/FRt2/assignment_loss‚3.4026378301473765 0loss/FBt1/assignment_loss‚0.0965006576452538 0loss/FBt3/detection_loss‚0.48122102329981026 $ _timestamp‚1.788473445050458e+09«æøˆŽÁ#‚(:ƒ ðàçÔÿ÷ˆ gpu.0.gpu‚12.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚137.36' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.34 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ا¬?…Â#‚(:ú ðàçÔüµ‡$ memory_percent‚38.23220729827881 cpu‚25.183110905472162 network.sent‚ 2.122557e+07 disk.md0p1.out‚ 1006.265625 disk.md0.in‚ 10.97265625# disk.nvme1n1.in‚201380.01953125 disk.dm-0.out‚ 118387.984375) proc.memory.percent‚18.50724220275879 network.recv‚ 3.251155e+06% disk./.usageGB‚2975.3438453674316 disk.md0p1.in‚ 10.97265625 disk.sdb1.in‚0* proc.memory.availableMB‚10120.03515625" proc.memory.rssMB‚ 3032.2265625 disk.sda1.out‚0 disk.sdb2.in‚5.40625" disk.sdb2.out‚1044.02294921875 disk.md0.out‚ 1006.265625 disk.rbd2.in‚0 disk.sda.in‚ 5.56640625" disk.nvme0n1.in‚329959.5078125" disk.sda2.out‚1044.02294921875 disk.sdb1.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda2.in‚ 5.56640625 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 2.80859375 disk.dm-0.in‚531339.52734375 disk.rbd2.out‚ 2.74609375# disk.nvme1n1.out‚62566.58203125! disk.sdb.out‚1044.02294921875# disk.nvme0n1.out‚55821.40234375( disk./.usagePercent‚83.1918591946157! disk.sda.out‚1044.02294921875 disk.sdb.in‚5.40625öø±‘Ã#‚(:† ÿàçÔíŠä… gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚140.478' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.1195 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÜŒ‰ÚƒÄ#‚(:ø ÿàçÔÿ÷¥‡ disk.rbd2.out‚ 2.74609375 cpu‚25.182848748564982 proc.cpu.threads‚17 network.sent‚ 2.1334828e+07 disk.rbd2.in‚0) proc.memory.percent‚18.50724220275879 disk.sdb.out‚1045.8486328125 disk.sdb1.out‚0# disk.nvme0n1.out‚55893.48046875 disk.md0p1.in‚ 10.97265625 disk.md0p1.out‚ 1008.0390625# disk.nvme1n1.out‚62669.97265625( proc.memory.availableMB‚ 10120.140625 disk.rbd0.in‚0.015625 network.recv‚ 3.257693e+06! disk.sda2.out‚1045.8486328125 disk.sda.out‚1045.8486328125 disk.dm-0.out‚ 118563.453125# disk.nvme1n1.in‚201380.01953125" proc.memory.rssMB‚ 3032.2265625) disk./.usagePercent‚83.19186143448688 disk.sdb2.in‚5.40625 disk.md0.in‚ 10.97265625! disk.sdb2.out‚1045.8486328125$ disk./.usageGB‚2975.343925476074 disk.sdb.in‚5.40625 disk.sdb1.in‚0 disk.sda.in‚ 5.56640625" disk.nvme0n1.in‚329959.5078125 disk.sda2.in‚ 5.56640625% memory_percent‚38.231563568115234 disk.dm-0.in‚531339.52734375 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0.out‚ 1008.0390625 disk.rbd0.out‚ 2.80859375‚(K‡…Å#‚(:ú ŽáçÔ¬êò†$ disk./.usageGB‚2975.343849182129" proc.memory.rssMB‚ 3032.2265625 proc.cpu.threads‚17 network.recv‚ 3.262369e+06 disk.rbd0.in‚0.015625! disk.sda.out‚1047.47021484375 disk.rbd0.out‚ 2.80859375 disk.sdb.in‚5.40625 disk.nvme1n1.out‚ 62747.53125 disk.md0p1.in‚ 10.97265625( proc.memory.availableMB‚ 10120.140625 network.sent‚ 2.1338903e+07) disk./.usagePercent‚83.19185930127624 disk.sdb2.in‚5.40625 disk.rbd2.out‚ 2.74609375# disk.nvme1n1.in‚201380.01953125! disk.sdb.out‚1047.47021484375 disk.sdb1.in‚0 disk.sda1.out‚0" disk.sdb2.out‚1047.47021484375 disk.sda.in‚ 5.56640625" disk.nvme0n1.in‚329959.5078125" disk.sda2.out‚1047.47021484375) proc.memory.percent‚18.50724220275879 disk.md0.in‚ 10.97265625 disk.sdb1.out‚0" disk.nvme0n1.out‚ 55965.5390625 disk.dm-0.in‚531339.52734375 disk.sda1.in‚0 disk.md0p1.out‚ 1009.60546875% memory_percent‚38.231563568115234 cpu‚25.18256445555652 disk.md0.out‚ 1009.60546875 disk.rbd2.in‚0 disk.dm-0.out‚118713.0703125 disk.sda2.in‚ 5.56640625ú?DzÆ#‚(:… ŽáçÔú͇ gpu.0.gpu‚26.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚146.048' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.512 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A°‹’Ç#‚(:‡ áçÔൣ† gpu.0.gpu‚24.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚142.313' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.57825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R7ãäÈ#‚(:ö áçÔîý“‡% disk./.usageGB‚2975.2737770080566# disk.nvme0n1.out‚56037.59765625 disk.dm-0.in‚531339.52734375 disk.rbd0.out‚ 2.80859375 disk.rbd2.in‚0# disk.nvme1n1.in‚201380.01953125 network.sent‚ 2.1346991e+07( disk./.usagePercent‚83.1899000539617 disk.sda2.in‚ 5.56640625! disk.sda2.out‚1048.8193359375 disk.md0p1.in‚ 10.97265625 disk.md0.in‚ 10.97265625 disk.md0.out‚ 1010.90625 disk.dm-0.out‚ 118940.140625 disk.sda1.out‚0 disk.rbd2.out‚ 2.74609375 disk.sdb.in‚5.40625 disk.sdb.out‚1048.8193359375" disk.nvme0n1.in‚329959.5078125 disk.sdb1.out‚0 disk.sda.in‚ 5.56640625 disk.sda.out‚1048.8193359375! disk.sdb2.out‚1048.8193359375% memory_percent‚38.231539726257324" proc.memory.rssMB‚ 3032.2265625 disk.rbd0.in‚0.015625* proc.memory.availableMB‚10120.14453125 cpu‚25.182277424546495# disk.nvme1n1.out‚62902.54296875 disk.sda1.in‚0 disk.md0p1.out‚ 1010.90625 disk.sdb2.in‚5.40625 disk.sdb1.in‚0) proc.memory.percent‚18.50724220275879 proc.cpu.threads‚17 network.recv‚ 3.266762e+06M~íÚ2É#- _wandbruntime‚9557  _runtime‚9557ºMÇÊ#Á % _timestamp‚1.7884735039022923e+09 _runtime‚9557.087249262 _step‚670 & lr-AdamW/pg1‚0.0002338967246506446 & lr-AdamW/pg2‚0.0002338967246506446 trainer/global_step‚8199 5ÌË#Æ _step‚670 & lr-AdamW/pg1‚0.0002338967246506446 & lr-AdamW/pg2‚0.0002338967246506446 trainer/global_step‚8199 % _timestamp‚1.7884735039022923e+09 _runtime‚9557.087249262žÑçF2Ì#- _wandbruntime‚9557  _runtime‚9557=ÿî'Í#èO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 301105152 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚317424067379536 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 155681887 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚369863215200256 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 155681885 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16285524575744 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚369863540245504 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚333528080006144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59102269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 152102463 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16239365254402 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚349813838250496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚333528062003200 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16560818484736 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 378472675 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325045248 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16561119589888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16285524575744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59101549 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 _step‚671 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349813586578944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚349813838250496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚349813586578944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 trainer/global_step‚8199 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16239594235486 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚333663432633938 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16285758244352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚333663678654382 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 _runtime‚9557.091630442 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 222790790 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 222817134 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚333528062003200 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚722 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 155681885 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 222817134 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚333528080006144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59102269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 152102463 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16239365254402 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚349813838250496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚333528062003200 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16560818484736 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 378472675 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325045248 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16561119589888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16285524575744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59101549 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 _step‚671 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349813586578944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚349813838250496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚349813586578944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 trainer/global_step‚8199 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16239594235486 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚333663432633938 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16285758244352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚333663678654382 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 _runtime‚9557.091630442 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 222790790 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 222817134 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚333528062003200 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚722 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 155681885 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 222817134 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 $ _timestamp‚1.788473505525934e+09 _runtime‚9558.712841521 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59108961 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 378544014 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 378544014 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚685 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16562832025088 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚333703167501000 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 152121362 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚333567366043136 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16241552724680 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 222817482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚317461597736960 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16241323743388 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚333567366043136 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 222817482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 222843852 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 378517642 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚93011716 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚349855088326656 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155700162 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 trainer/global_step‚8199 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 152120677 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 378517642 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93011718 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 222843852 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16287722283520 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚349854836641792©FÒü&Ñ#öM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 378544014 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚685 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16562832025088 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚333703167501000 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 152121362 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚333567366043136 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16241552724680 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 222817482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚317461597736960 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16241323743388 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚333567366043136 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 222817482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 222843852 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 378517642 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚93011716 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚349855088326656 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155700162 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 trainer/global_step‚8199 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 152120677 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 378517642 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93011718 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 222843852 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16287722283520 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚349854836641792 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16287722283520 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16287488601600 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 $ _timestamp‚1.788473505525934e+09 _runtime‚9558.712841521 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59108961 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 378544014 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 êÕ E2Ò#- _wandbruntime‚9558  _runtime‚9558.\a]Ó#× /loss/FBt1/detection_loss‚3.8465450406074524 /loss/FBt2/detection_loss‚1.5428716430538578 %loss/total_loss‚41.34578854439385 epoch‚6 _runtime‚9558.714372057 2loss/SRqqt4/assignment_loss‚1.3416832699017092 0loss/FRt2/assignment_loss‚2.8225486654302347 2loss/SRqqt1/assignment_loss‚1.5626185903031575 1loss/SRqqt1/detection_loss‚2.0849935733958294 0loss/SRqqt2/detection_loss‚2.516477490464846 /loss/FBt4/detection_loss‚2.0319012148039683 /loss/FRt3/assignment_loss‚1.282295222243955 -loss/FRt3/detection_loss‚1.54849834211411 1loss/SRqqt4/detection_loss‚2.6894274814562364 trainer/global_step‚8199 % _timestamp‚1.7884735055294423e+09 1loss/FBt2/assignment_loss‚0.16335736851364777 2loss/FBt3/assignment_loss‚0.002450626001423828 1loss/FBt4/assignment_loss‚0.24531923442345974 .loss/FRt1/detection_loss‚2.617750741979655 _step‚673 /loss/FRt1/assignment_loss‚2.818814707152984 0loss/FRt4/assignment_loss‚3.5225032046437263 2loss/SRqqt3/assignment_loss‚0.7884056077230884 0loss/FBt1/assignment_loss‚0.3066554423418893 1loss/SRqqt3/detection_loss‚0.8445447207932625 /loss/FBt3/detection_loss‚0.6715904582447062 2loss/SRqqt2/assignment_loss‚1.2880638102069497 /loss/FRt2/detection_loss‚1.9365591159199222 /loss/FRt4/detection_loss‚2.8699129726737738Ê#ãbÔ#Ü /loss/FBt4/detection_loss‚2.0319012148039683 /loss/FRt3/assignment_loss‚1.282295222243955 -loss/FRt3/detection_loss‚1.54849834211411 1loss/SRqqt4/detection_loss‚2.6894274814562364 trainer/global_step‚8199 % _timestamp‚1.7884735055294423e+09 1loss/FBt2/assignment_loss‚0.16335736851364777 2loss/FBt3/assignment_loss‚0.002450626001423828 1loss/FBt4/assignment_loss‚0.24531923442345974 .loss/FRt1/detection_loss‚2.617750741979655 _step‚673 /loss/FRt1/assignment_loss‚2.818814707152984 0loss/FRt4/assignment_loss‚3.5225032046437263 2loss/SRqqt3/assignment_loss‚0.7884056077230884 0loss/FBt1/assignment_loss‚0.3066554423418893 1loss/SRqqt3/detection_loss‚0.8445447207932625 /loss/FBt3/detection_loss‚0.6715904582447062 2loss/SRqqt2/assignment_loss‚1.2880638102069497 /loss/FRt2/detection_loss‚1.9365591159199222 /loss/FRt4/detection_loss‚2.8699129726737738 /loss/FBt1/detection_loss‚3.8465450406074524 /loss/FBt2/detection_loss‚1.5428716430538578 %loss/total_loss‚41.34578854439385 epoch‚6 _runtime‚9558.714372057 2loss/SRqqt4/assignment_loss‚1.3416832699017092 0loss/FRt2/assignment_loss‚2.8225486654302347 2loss/SRqqt1/assignment_loss‚1.5626185903031575 1loss/SRqqt1/detection_loss‚2.0849935733958294 0loss/SRqqt2/detection_loss‚2.516477490464846¡:¤Ö‘Õ#‚(:† ¬áçÔ‡ž›ˆ gpu.0.gpu‚28.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚104.494' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚26.1235 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00Fê€Ö#‚(:õ ¬áçÔ°ç°‰ disk.md0p1.in‚ 10.97265625 disk.sdb.in‚5.40625 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 10120.1171875 disk.sdb1.in‚0! disk.sda.out‚1051.00927734375 disk.dm-0.out‚ 119055.96875 disk.md0.in‚ 10.97265625 disk.rbd0.out‚ 2.80859375 network.sent‚ 2.135079e+07 disk.md0.out‚ 1013.03125 disk.rbd2.in‚0# disk.nvme1n1.in‚201380.01953125 disk.sdb1.out‚0 disk.sda.in‚ 5.56640625 proc.cpu.threads‚17" disk.sdb2.out‚1051.00927734375 network.recv‚ 3.27095e+06" disk.sda2.out‚1051.00927734375# disk.nvme1n1.out‚62946.29296875) proc.memory.percent‚18.50724220275879 disk.dm-0.in‚531339.52734375 disk.sdb2.in‚5.40625" disk.nvme0n1.in‚329959.5078125% memory_percent‚38.231706619262695" proc.memory.rssMB‚ 3032.2265625 cpu‚25.182006560614518! disk.sdb.out‚1051.00927734375( disk./.usagePercent‚83.1771081496838$ disk./.usageGB‚2974.816276550293 disk.sda2.in‚ 5.56640625 disk.md0p1.out‚ 1013.03125 disk.rbd2.out‚ 2.74609375# disk.nvme0n1.out‚56109.67578125 disk.rbd0.in‚0.015625t5ÚP×#‚(:„ »áçԠл‡ gpu.0.gpu‚8.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚100.15' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.0375 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÛÙ0‰Ø#‚(:þ »áçÔ¡ž£ˆ network.recv‚ 3.277493e+06 disk.sdb.in‚5.40625 disk.dm-0.in‚ 531339.578125 disk.sda1.in‚0% memory_percent‚38.231682777404785" proc.memory.rssMB‚ 3032.2265625 proc.cpu.threads‚17 network.sent‚ 2.145992e+07 disk.sdb1.out‚0# disk.nvme0n1.out‚56181.72265625 disk.md0.in‚ 10.97265625 disk.rbd0.in‚0.015625 disk.rbd2.in‚0# disk.nvme1n1.in‚201380.01953125# disk.nvme1n1.out‚62946.94921875! disk.sdb.out‚1052.41064453125 disk.sda2.in‚ 5.56640625" disk.sda2.out‚1052.41064453125 disk.md0p1.in‚ 10.97265625* proc.memory.availableMB‚10120.12109375% disk./.usageGB‚2974.8068618774414 disk.md0p1.out‚ 1014.38671875 disk.sdb2.in‚5.40625" disk.sdb2.out‚1052.41064453125 disk.rbd2.out‚ 2.74609375) proc.memory.percent‚18.50724220275879 cpu‚25.18178076156771) disk./.usagePercent‚83.17684491149053 disk.md0.out‚ 1014.38671875 disk.sdb1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚ 2.80859375 disk.sda.in‚ 5.56640625! disk.sda.out‚1052.41064453125# disk.nvme0n1.in‚329959.55859375 disk.dm-0.out‚ 119128.671875o­‘Ù#‚(:† ÊáçÔûƇ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚136.466' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.1165 gpu.0.smClock‚1260 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©PJ‰Ú#‚(:þ ÊáçÔË‹­ˆ disk.dm-0.in‚ 531339.578125 disk.md0.out‚ 1015.89453125 cpu‚25.18157514991335 disk.rbd0.in‚0.015625 disk.sda1.out‚0 disk.sda2.in‚ 5.56640625 disk.rbd2.out‚ 2.74609375# disk.nvme1n1.out‚63021.08203125! disk.dm-0.out‚119274.93359375 disk.sda1.in‚0) disk./.usagePercent‚83.17788069191796# disk.nvme1n1.in‚201380.01953125# disk.nvme0n1.in‚329959.55859375 disk.md0p1.in‚ 10.97265625 disk.md0.in‚ 10.97265625 proc.cpu.threads‚17 network.recv‚ 3.282301e+06 disk.sdb.in‚5.40625 disk.sdb1.in‚0 disk.sdb1.out‚0" disk.nvme0n1.out‚ 56253.8515625% memory_percent‚38.233232498168945) proc.memory.availableMB‚ 10119.8671875" proc.memory.rssMB‚ 3032.2265625$ disk./.usageGB‚2974.843906402588 disk.rbd0.out‚ 2.80859375 disk.rbd2.in‚0 disk.sda.in‚ 5.56640625! disk.sdb.out‚1053.97216796875! disk.sda.out‚1053.97216796875" disk.sda2.out‚1053.97216796875) proc.memory.percent‚18.50724220275879 network.sent‚ 2.1464096e+07 disk.md0p1.out‚ 1015.89453125 disk.sdb2.in‚5.40625" disk.sdb2.out‚1053.97216796875–2ÈÕ‘Û#‚(:† ÙáçÔ©Á‡ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚95.529' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚23.88225 gpu.0.smClock‚1200 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öXÜ#‚(:ù ÙáçÔÙèˆ disk.sda2.in‚ 5.56640625 disk.md0.out‚ 1017.28515625" disk.nvme1n1.out‚ 63098.6796875# disk.nvme0n1.in‚329959.55859375" disk.nvme0n1.out‚ 56325.9296875 disk.md0p1.in‚ 10.97265625 cpu‚25.181318972295266 proc.cpu.threads‚17 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 2.80859375 disk.rbd2.out‚ 2.74609375# disk.nvme1n1.in‚201380.01953125 disk.sdb.in‚5.40625 disk.sda1.out‚0! disk.sda2.out‚1055.4072265625) proc.memory.percent‚18.50724220275879! disk.sdb2.out‚1055.4072265625 disk.md0.in‚ 10.97265625 disk.rbd2.in‚0 disk.sdb.out‚1055.4072265625" proc.memory.rssMB‚ 3032.2265625 network.sent‚ 2.1472293e+07 disk.sdb1.out‚0 disk.sda.in‚ 5.56640625 disk.sda.out‚1055.4072265625) proc.memory.availableMB‚ 10119.8671875) disk./.usagePercent‚83.18137403766602% disk./.usageGB‚2974.9688453674316 disk.dm-0.in‚ 531339.578125 disk.dm-0.out‚ 119424.609375 disk.md0p1.out‚ 1017.28515625% memory_percent‚38.233232498168945 network.recv‚ 3.286621e+06 disk.sdb2.in‚5.40625 disk.sdb1.in‚0 disk.sda1.in‚0y J¡2Ý#- _wandbruntime‚9619  _runtime‚9619YZFJÉÞ#à ' lr-AdamW/pg1‚0.00022855019700398683 ' lr-AdamW/pg2‚0.00022855019700398683 trainer/global_step‚8249 % _timestamp‚1.7884735662772713e+09 _runtime‚9619.462480953 _step‚674wÕ…RÎß#È trainer/global_step‚8249 % _timestamp‚1.7884735662772713e+09 _runtime‚9619.462480953 _step‚674 ' lr-AdamW/pg1‚0.00022855019700398683 ' lr-AdamW/pg2‚0.00022855019700398683¢=6·­2à#- _wandbruntime‚9619  _runtime‚9619 mem§ á#èO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 224128742 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚93541016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 224155086 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16337532275994 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚351876104135168 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚351876355806720 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156595635 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚319300602290096 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚351876355806720 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 224128742 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 152990512 H?DeviceStatsMoniw›åmGtor.on_train_batch_start/inactive_split.all.peak‚1319 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚355386595217920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚319300585250736 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16337303294910 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 152991207 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156595637 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚335492363852800 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚351876104135168 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚335492381855744 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚335492363852800 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16383740282368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚8249 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚355386619158016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚335637888545646 % _timestamp‚1.7884735662785125e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59450191 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 380750723 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚335638134566090 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 _step‚675 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16383740282368 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16660781293568 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚372047705751040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 224155086 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚695 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 380724377 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚693 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16383973950976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 _runtime‚9619.463880879 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚335492381855744 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93541014 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16661086593024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 156595637 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59449498 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚351876104135168 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚335492381855744 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚335492363852800 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16383740282368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚8249 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚355386619158016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚335637888545646 % _timestamp‚1.7884735662785125e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59450191 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 380750723 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚335638134566090 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 _step‚675 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16383740282368 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16660781293568 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚372047705751040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 224155086 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚695 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 380724377 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚693 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16383973950976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 _runtime‚9619.463880879 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚335492381855744 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93541014 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16661086593024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 156595637 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59449498 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚351876355806720 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156595635 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚319300602290096 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚351876355806720 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 224128742 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 152990512£9Å}62ã#- _wandbruntime‚9620  _runtime‚9620|ä öö&ä#ðM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚319338115608160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _runtime‚9620.668484888 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚372091063949312 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚335531667892736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16385704308224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 380795716 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 380769344 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚351917605882880 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚372091370106880 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59456102 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16663066978816 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚355428303128064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚335531649889792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚351917605882880 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16662784761344 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚335677623412708 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚335677377392056 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚319338132647520 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16385704308224 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153007918 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16385937990144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 380795716 _step‚676 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 224181804 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59456757 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156613910 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156613912 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16339261783896 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153008575 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚657 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 224155434 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚335531649889792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93551818 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚335531667892736 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16339490765188 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309]åÖ {å#õM FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 224155434 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚335531649889792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93551818 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚335531667892736 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16339490765188 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚319338115608160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _runtime‚9620.668484888 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚372091063949312 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚335531667892736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16385704308224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 380795716 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 380769344 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚351917605882880 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚372091370106880 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59456102 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16663066978816 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚355428303128064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/Xv~@€active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚335531649889792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚351917605882880 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16662784761344 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚335677623412708 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚335677377392056 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚319338132647520 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16385704308224 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153007918 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16385937990144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 380795716 _step‚676 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 224181804 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59456757 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156613910 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156613912 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16339261783896 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153008575 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚657 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚353938873362944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚337456665702400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _runtime‚9681.746678041 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16481955988992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚337612344457354 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304073728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚337456683705344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 383002425 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280133632 _step‚679 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚353938621691392 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 153879937 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16760594576896 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚697 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 157509385 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚374231692370944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚337612590477798 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚374231388297216 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚353938873362944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚337456665702400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _runtime‚9681.746678041 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16481955988992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚337612344457354 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304073728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚337456683705344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 383002425 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280133632 _step‚679 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚353938621691392 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 153879937 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16760594576896 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚697§êX(á2÷#- _wandbruntime‚9682  _runtime‚9682seÇ/ø&ø#òM >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚94091918 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 225519756 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59804751 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16437428805696 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 157527660 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153897347 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚357512477690368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚353979871754240 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 383047418 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 383021046 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 157527662 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16437199824404 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚353980123439104 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚337495951739392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚337495969742336 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16484153696768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚337652079324416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚321214633479360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚337651833303764 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚374275062509568 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59805429 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16762875425280 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16762584819200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153896667 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 157527662 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚357512501630464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 383021046 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16483920014848 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 % _timestamp‚1.7884736295392606e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚353980123439104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚337495951739392 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 225493386 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 225519756 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚8299 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16484153696768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚337495969742336 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496Ž™°1ý&ù#÷M J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚353980123439104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚337495951739392 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 225493386 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 225519756 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚8299 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16484153696768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚337495969742336 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚94091918 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 225519756 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59804751 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16437428805696 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 157527660 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153897347 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚357512477690368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚353979871754240 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 383047418 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 383021046 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 157527662 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16437199824404 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚353980123439104 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚337495951739392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚337495969742336 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16484153696768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚337652079324416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚321214633479360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚337651833303764 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚374275062509568 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59805429 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16762875425280 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16762584819200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153896667 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 157527662 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚357512501630464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 383021046 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16483920014848 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 % _timestamp‚1.7884736295392606e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652¨h¬dŸ2ú#- _wandbruntime‚9682  _runtime‚96828áý[û#Õ % _timestamp‚1.7884736295415347e+09 0loss/FRt3/assignment_loss‚0.9627172946929932 0loss/SRqqt4/detection_loss‚2.791871177879247 0loss/FBt3/detection_loss‚0.48611189106322195 /loss/FRt2/assignment_loss‚3.149269163608551 /loss/FRt1/detection_loss‚2.6862075857577787 1loss/SRqqt2/detection_loss‚2.3663292044195634 _runtime‚9682.726222307 2loss/SRqqt1/assignment_loss‚1.4328458742579309 1loss/SRqqt3/assignment_loss‚0.854846119850708 0loss/FBt4/assignment_loss‚0.5169982633983287 .loss/FRt4/detection_loss‚3.242403003076712 0loss/SRqqt1/detection_loss‚1.918037705305146 -loss/FBt4/detection_loss‚2.24005879347141 epoch‚6 trainer/global_step‚8299 1loss/FBt1/assignment_loss‚0.17055231268463664 .loss/FRt3/detection_loss‚1.378911990644128 3loss/FBt3/assignment_loss‚0.0077046955901648225 .loss/FBt1/detection_loss‚2.036707648864159 _step‚681 1loss/SRqqt2/assignment_loss‚1.531199826762594 2loss/SRqqt4/assignment_loss‚1.4891271889209747 /loss/FRt2/detection_loss‚2.0835748441382558 /loss/FBt2/detection_loss‚1.8966048061847687 %loss/total_loss‚38.93084454277892 0loss/FRt1/assignment_loss‚2.3607416499045586 0loss/FRt4/assignment_loss‚2.5732132866978645 1loss/FBt2/assignment_loss‚0.10506768819217882 1loss/SRqqt3/detection_loss‚0.6497425274130436ÛìÞý`ü#Ú 1loss/FBt1/assignment_loss‚0.17055231268463664 .loss/FRt3/detection_loss‚1.378911990644128 3loss/FBt3/assignment_loss‚0.0077046955901648225 .loss/FBt1/detection_loss‚2.036707648864159 _step‚681 1loss/SRqqt2/assignment_loss‚1.531199826762594 2loss/SRqqt4/assignment_loss‚1.4891271889209747 /loss/FRt2/detection_loss‚2.0835748441382558 /loss/FBt2/detection_loss‚1.8966048061847687 %loss/total_loss‚38.93084454277892 0loss/FRt1/assignment_loss‚2.3607416499045586 0loss/FRt4/assignment_loss‚2.5732132866978645 1loss/FBt2/assignment_loss‚0.10506768819217882 1loss/SRqqt3/detection_loss‚0.6497425274130436 % _timestamp‚1.7884736295415347e+09 0loss/FRt3/assignment_loss‚0.9627172946929932 0loss/SRqqt4/detection_loss‚2.791871177879247 0loss/FBt3/detection_loss‚0.48611189106322195 /loss/FRt2/assignment_loss‚3.149269163608551 /loss/FRt1/detection_loss‚2.6862075857577787 1loss/SRqqt2/detection_loss‚2.3663292044195634 _runtime‚9682.726222307 2loss/SRqqt1/assignment_loss‚1.4328458742579309 1loss/SRqqt3/assignment_loss‚0.854846119850708 0loss/FBt4/assignment_loss‚0.5169982633983287 .loss/FRt4/detection_loss‚3.242403003076712 0loss/SRqqt1/detection_loss‚1.918037705305146 -loss/FBt4/detection_loss‚2.24005879347141 epoch‚6 trainer/global_step‚8299©é²¿‘ý#‚(:† ¤âçÔç°‘ˆ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚104.826' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚26.2065 gpu.0.smClock‚1200 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í¤²“{þ#‚(:ð ¤âçÔáŽùˆ disk.sda1.out‚0# disk.nvme1n1.out‚63559.35546875 disk.md0.in‚ 10.97265625 disk.md0.out‚ 1025.94140625 disk.sdb.in‚5.40625 disk.md0p1.out‚ 1025.94140625" proc.memory.rssMB‚ 3032.2265625# disk.nvme0n1.in‚329959.55859375 disk.md0p1.in‚ 10.97265625) proc.memory.percent‚18.50724220275879 network.recv‚ 3.309074e+06 disk.rbd0.in‚0.015625 disk.sdb1.out‚0 disk.dm-0.out‚ 120246.46875 disk.rbd2.out‚ 2.74609375 disk.sda.out‚1064.365234375# disk.nvme0n1.out‚56687.11328125 disk.sda2.in‚ 5.56640625 disk.sda2.out‚1064.365234375 disk.sdb2.out‚1064.365234375 disk.rbd0.out‚ 2.80859375 disk.sdb.out‚1064.365234375 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.dm-0.in‚531343.05859375$ memory_percent‚38.23525905609131 proc.cpu.threads‚17% disk./.usageGB‚2974.8161964416504 disk.nvme1n1.in‚201383.5 network.sent‚ 2.160037e+07) disk./.usagePercent‚83.17710590981264 disk.sdb2.in‚5.40625 disk.sda1.in‚0* proc.memory.availableMB‚10119.53515625 cpu‚25.180174075584972 disk.sda.in‚ 5.56640625Ȳ_‘ÿ#‚(:† ³âçÔ²ô‡ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚91.795' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚22.94875 gpu.0.smClock‚1170 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹¹ ©€$‚(:ö ³âçÔèåˆ) disk./.usagePercent‚83.17586470120033 disk.rbd0.in‚0.015625 disk.sdb1.in‚0$ memory_percent‚38.23525905609131 cpu‚25.17995237699147 disk.dm-0.out‚120363.1015625 disk.sda2.in‚ 5.56640625" disk.sda2.out‚1065.97607421875* proc.memory.availableMB‚10119.53515625 network.sent‚ 2.1707194e+07 disk.rbd2.in‚0 disk.rbd2.out‚ 2.74609375% disk./.usageGB‚2974.7718048095703! disk.sda.out‚1065.97607421875 disk.nvme0n1.in‚ 329972.65625 disk.dm-0.in‚531380.8828125 disk.md0p1.out‚ 1027.48046875 disk.sdb2.in‚5.40625 disk.md0.in‚ 10.97265625 disk.md0.out‚ 1027.48046875" disk.nvme1n1.out‚ 63602.9765625 disk.sdb.in‚5.40625 proc.cpu.threads‚17 network.recv‚ 3.315749e+06 disk.nvme0n1.out‚ 56760.125 disk.sda1.in‚0" disk.nvme1n1.in‚201408.2265625) proc.memory.percent‚18.50724220275879 disk.sda.in‚ 5.56640625 disk.md0p1.in‚ 10.97265625" disk.sdb2.out‚1065.97607421875 disk.rbd0.out‚ 2.80859375! disk.sdb.out‚1065.97607421875" proc.memory.rssMB‚ 3032.2265625 disk.sda1.out‚0 disk.sdb1.out‚0]žŽ$‚(:… ÂâçÔÕÞó‡ gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚32.0 gpu.0.powerWatts‚91.186' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚22.7965 gpu.0.smClock‚1200 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*UÓt‚‚$‚(:÷ ÂâçÔ‡Ó݈ disk.rbd0.out‚ 2.80859375 network.recv‚ 3.320465e+06 disk.md0p1.out‚ 1028.93359375! disk.sda.out‚1067.49755859375! disk.sdb.out‚1067.49755859375 disk.sdb1.out‚0# disk.nvme0n1.out‚56832.89453125" proc.memory.rssMB‚ 3032.2265625 proc.cpu.threads‚17" disk.sdb2.out‚1067.49755859375 disk.md0.in‚ 10.97265625" disk.nvme1n1.in‚201408.2265625 disk.sdb.in‚5.40625 disk.sda.in‚ 5.56640625 network.sent‚ 2.1713697e+07 disk.sdb1.in‚0 disk.nvme0n1.in‚ 329972.65625! disk.dm-0.out‚120508.36328125 disk.sda1.out‚0' proc.memory.availableMB‚ 10119.53125) proc.memory.percent‚18.50724220275879 cpu‚25.17972358444732) disk./.usagePercent‚83.17781189587474 disk.md0p1.in‚ 10.97265625 disk.nvme1n1.out‚ 63675.46875 disk.sda2.in‚ 5.56640625" disk.sda2.out‚1067.49755859375% disk./.usageGB‚2974.8414459228516 disk.md0.out‚ 1028.93359375 disk.sda1.in‚0 disk.rbd2.out‚ 2.74609375 disk.sdb2.in‚5.40625 disk.rbd0.in‚0.015625$ memory_percent‚38.23528289794922 disk.rbd2.in‚0 disk.dm-0.in‚531380.8828125 ”Û¼‘ƒ$‚(:† ÑâçÔÞƒ¯‡ gpu.0.gpu‚38.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚107.002' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚26.7505 gpu.0.smClock‚1230 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äSE“{„$‚(:ð ÑâçÔœ±½ˆ disk.sda.in‚ 5.56640625) proc.memory.percent‚18.50724220275879 disk.sdb.in‚5.40625" proc.memory.rssMB‚ 3032.2265625 cpu‚25.179525740004113 proc.cpu.threads‚17 disk.sdb2.in‚5.40625 disk.md0.out‚ 1030.73828125 disk.sdb1.out‚0) disk./.usagePercent‚83.17962235174699 disk.sda2.in‚ 5.56640625 disk.md0p1.out‚ 1030.73828125 disk.rbd0.out‚ 2.80859375 disk.nvme1n1.out‚ 63703.59375 disk.sda.out‚1069.3583984375 network.sent‚ 2.1721743e+07! disk.sda2.out‚1069.3583984375 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.sdb1.in‚0' proc.memory.availableMB‚ 10119.53125! disk.nvme0n1.out‚ 56905.359375 disk.dm-0.in‚531380.8828125 disk.dm-0.out‚ 120608.953125 disk.md0.in‚ 10.97265625 disk.sdb.out‚1069.3583984375 disk.sda1.out‚0 disk.md0p1.in‚ 10.97265625 disk.rbd2.out‚ 2.74609375$ memory_percent‚38.23528289794922 network.recv‚ 3.324676e+06% disk./.usageGB‚2974.9061965942383 disk.nvme0n1.in‚ 329972.65625 disk.sda1.in‚0! disk.sdb2.out‚1069.3583984375" disk.nvme1n1.in‚201408.2265625ŽØv2…$- _wandbruntime‚9743  _runtime‚9743B O°Ç†$Á & lr-AdamW/pg2‚0.0002179730814475981 trainer/global_step‚8349 % _timestamp‚1.7884736905337026e+09 _runtime‚9743.718668829 _step‚682 & lr-AdamW/pg1‚0.0002179730814475981nØœÇ̇$Æ & lr-AdamW/pg1‚0.0002179730814475981 & lr-AdamW/pg2‚0.0002179730814475981 trainer/global_step‚8349 % _timestamp‚1.7884736905337026e+09 _runtime‚9743.718668829 _step‚682ª ,Y2ˆ$- _wandbruntime‚9743  _runtime‚9743Š k@3‰$èO LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16580171695616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 226830990 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 226830990 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 158423137 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 158423135 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 226804646 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitorç \þ»".on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚356001390919168 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 154761999 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16580405364224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 158423137 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚692 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚376415444096512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16580405364224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚60140783 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16533179375926 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 226804646 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚323053620993136 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60140093 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚359554992222720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚339586800369062 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16860740396032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158423135 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚339420985554944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚339587046389506 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚359555016162816 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16860451873792 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚94621216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 % _timestamp‚1.7884736905347674e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚339420967552000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 385227781 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚356001139247616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚356001390919168 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 385254127 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 trainer/global_step‚8349 _step‚683 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚323053620993136 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60140093 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚359554992222720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚339586800369062 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16860740396032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158423135 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚339420985554944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚339587046389506 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚359555016162816 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16860451873792 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚94621216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 % _timestamp‚1.7884736905347674e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚339420967552000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 385227781 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚356001139247616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚356001390919168 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 385254127 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 trainer/global_step‚8349 _step‚683 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 385299120 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚356042640995328 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154778827 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚359596700132864 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚339626535236124 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 226831338 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 158441410 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚94632018 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60146811 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16535137864912 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16535366846204 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 385299120 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 226857708 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚339460253588992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 226831338 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚687 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚685 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 # _timestamp‚1.78847369167363e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60147496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚356042389310464 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16582135721472 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 trainer/global_step‚8349 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚376459435993600 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 158441412 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚339460271591936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚339626289215472 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 385272748÷´ÉX$õM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 385299120 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚356042640995328 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154778827 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚359596700132864 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚339626535236124 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 226831338 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 158441410 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚94632018 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60146811 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16535137864912 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16535366846204 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 385299120 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 226857708 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚339460253588992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 226831338 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚687 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚685 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 # _timestamp‚1.78847369167363e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60147496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚356042389310464 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16582135721472 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 trainer/global_step‚8349 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚376459435993600 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 158441412 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚339460271591936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚339626289215472 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 385272748 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 226857708 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚94632016 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608¬ÔñKâ2Ž$- _wandbruntime‚9744  _runtime‚9744hsÍŠ[$Õ /loss/FRt4/assignment_loss‚3.284158829380484 /loss/FRt2/detection_loss‚1.9262005253023324 /loss/FRt3/detection_loss‚1.8240072366781532 epoch‚6 % _timestamp‚1.7884736916765122e+09 /loss/FRt1/assignment_loss‚2.754430380306746 0loss/FRt3/assignment_loss‚1.5204771091540654 2loss/SRqqt3/assignment_loss‚1.0498247966262522 trainer/global_step‚8349 0loss/FRt2/assignment_loss‚3.3211156783195643 1loss/FBt3/assignment_loss‚0.06338498841267598 1loss/SRqqt3/detection_loss‚0.8197302015701143 /loss/FBt3/detection_loss‚0.6378270838056402 2loss/SRqqt4/assignment_loss‚1.6183916528318678 1loss/FBt4/assignment_loss‚0.28634030850656583 0loss/SRqqt2/detection_loss‚2.262823775410652 /loss/FBt2/detection_loss‚2.0523208550044467 /loss/FBt4/detection_loss‚1.9551118264595668 _step‚685 0loss/FBt2/assignment_loss‚0.2039067972172656 2loss/SRqqt1/assignment_loss‚1.2351892376091422 1loss/SRqqt4/detection_loss‚2.2762013808373482 &loss/total_loss‚40.484186365983334 _runtime‚9744.864548387 /loss/FBt1/assignment_loss‚0.184396772586248 /loss/FRt1/detection_loss‚2.6135617416155967 .loss/SRqqt1/detection_loss‚2.0835026618911 2loss/SRqqt2/assignment_loss‚1.2357596119537073 /loss/FRt4/detection_loss‚2.7859564861830544 /loss/FBt1/detection_loss‚2.4895664283207486<=Hû`$Ú 1loss/SRqqt3/detection_loss‚0.8197302015701143 /loss/FBt3/detection_loss‚0.6378270838056402 2loss/SRqqt4/assignment_loss‚1.6183916528318678 1loss/FBt4/assignment_loss‚0.28634030850656583 0loss/SRqqt2/detection_loss‚2.262823775410652 /loss/FBt2/detection_loss‚2.0523208550044467 /loss/FBt4/detection_loss‚1.9551118264595668 _step‚685 0loss/FBt2/assignment_loss‚0.2039067972172656 2loss/SRqqt1/assignment_loss‚1.2351892376091422 1loss/SRqqt4/detection_loss‚2.2762013808373482 &loss/total_loss‚40.484186365983334 _runtime‚9744.864548387 /loss/FBt1/assignment_loss‚0.184396772586248 /loss/FRt1/detection_loss‚2.6135617416155967 .loss/SRqqt1/detection_loss‚2.0835026618911 2loss/SRqqt2/assignment_loss‚1.2357596119537073 /loss/FRt4/detection_loss‚2.7859564861830544 /loss/FBt1/detection_loss‚2.4895664283207486 /loss/FRt4/assignment_loss‚3.284158829380484 /loss/FRt2/detection_loss‚1.9262005253023324 /loss/FRt3/detection_loss‚1.8240072366781532 epoch‚6 % _timestamp‚1.7884736916765122e+09 /loss/FRt1/assignment_loss‚2.754430380306746 0loss/FRt3/assignment_loss‚1.5204771091540654 2loss/SRqqt3/assignment_loss‚1.0498247966262522 trainer/global_step‚8349 0loss/FRt2/assignment_loss‚3.3211156783195643 1loss/FBt3/assignment_loss‚0.06338498841267598­ê¸EÐ’‘$‚(:‡ àâçÔóħ‡ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚102.935' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.73375 gpu.0.smClock‚1185 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03~D?s’$‚(:è àâçÔ¯ïñˆ disk.rbd0.in‚0.015625 disk.rbd2.out‚ 2.74609375 disk.sda1.out‚0 disk.sdb.in‚5.40625 disk.sdb1.in‚0$ memory_percent‚38.23566436767578$ disk./.usageGB‚2975.094135284424 disk.md0p1.out‚ 1032.26171875 disk.md0.out‚ 1032.26171875 disk.sda1.in‚0 disk.md0p1.in‚ 10.97265625 disk.sdb.out‚1070.943359375) proc.memory.percent‚18.50724220275879 disk.sdb1.out‚0 disk.sda2.out‚1070.943359375 disk.rbd0.out‚ 2.80859375 disk.nvme0n1.in‚ 329972.65625 disk.md0.in‚ 10.97265625! disk.nvme1n1.out‚ 63755.546875' proc.memory.availableMB‚ 10119.46875 proc.cpu.threads‚17 disk.dm-0.out‚ 120733.375 disk.sda.in‚ 5.56640625 disk.sda.out‚1070.943359375! disk.nvme0n1.out‚ 56977.828125 cpu‚25.17927381649227 disk.dm-0.in‚531380.8828125 disk.sda2.in‚ 5.56640625 disk.sdb2.out‚1070.943359375 disk.rbd2.in‚0" proc.memory.rssMB‚ 3032.2265625" disk.nvme1n1.in‚201408.2265625 network.sent‚ 2.1738362e+07 network.recv‚ 3.328822e+06) disk./.usagePercent‚83.18487719618301 disk.sdb2.in‚5.40625ª­–Ú‘“$‚(:† ïâçÔïÇà‡ gpu.0.gpu‚37.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚104.826' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚26.2065 gpu.0.smClock‚1230 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XÅP}”$‚(:ò ïâçÔ°Ýň" disk.sdb2.out‚1072.69384765625 disk.rbd0.in‚0.015625 disk.sdb1.in‚0 proc.cpu.threads‚17 disk.rbd2.out‚ 2.74609375 disk.sda.in‚ 5.56640625! disk.sda.out‚1072.69384765625' proc.memory.availableMB‚ 10119.46875 network.recv‚ 3.33546e+06 disk.dm-0.out‚ 120934.984375$ memory_percent‚38.23566436767578 network.sent‚ 2.183245e+07 disk.dm-0.in‚531380.8984375 disk.sda1.in‚0 disk.sda1.out‚0" disk.sda2.out‚1072.69384765625 disk.sdb2.in‚5.40625 disk.md0.in‚ 10.97265625 disk.md0.out‚ 1033.953125# disk.nvme1n1.out‚63884.76171875 disk.sdb.in‚5.40625! disk.sdb.out‚1072.69384765625) proc.memory.percent‚18.50724220275879 cpu‚25.17905228470252 disk.md0p1.in‚ 10.97265625" disk.nvme1n1.in‚201408.2421875 disk.nvme0n1.in‚ 329972.65625# disk.nvme0n1.out‚57050.22265625 disk.md0p1.out‚ 1033.953125 disk.sdb1.out‚0% disk./.usageGB‚2975.3447074890137 disk.sda2.in‚ 5.56640625 disk.rbd0.out‚ 2.80859375 disk.rbd2.in‚0" proc.memory.rssMB‚ 3032.2265625) disk./.usagePercent‚83.19188329989596™Bdí’•$‚(:‡ þâçÔóÑᇠgpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚102.935' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.73375 gpu.0.smClock‚1200 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Å-Joz–$‚(:ï þâçÔÏžˆ network.recv‚ 3.340048e+06 disk.md0.in‚ 10.97265625 disk.rbd0.in‚0.015625$ memory_percent‚38.23566436767578' proc.memory.availableMB‚ 10119.46875) proc.memory.percent‚18.50724220275879 disk.sdb2.in‚5.40625" disk.nvme1n1.in‚201408.2421875 disk.sdb.in‚5.40625 disk.sdb.out‚1074.6630859375 disk.sda1.in‚0 proc.cpu.threads‚17 disk.md0.out‚ 1035.859375 disk.rbd2.out‚ 2.74609375 disk.sda.out‚1074.6630859375 disk.nvme0n1.in‚ 329972.65625 disk.dm-0.in‚531380.8984375 disk.sda2.in‚ 5.56640625! disk.sda2.out‚1074.6630859375 disk.md0p1.out‚ 1035.859375 network.sent‚ 2.1841094e+07 disk.rbd2.in‚0 disk.nvme1n1.out‚ 63988.3125 disk.sdb1.in‚0) disk./.usagePercent‚83.19185439489176 disk.sda.in‚ 5.56640625! disk.dm-0.out‚121110.82421875% disk./.usageGB‚2975.3436737060547 disk.rbd0.out‚ 2.80859375# disk.nvme0n1.out‚57122.51171875 disk.sda1.out‚0! disk.sdb2.out‚1074.6630859375 disk.sdb1.out‚0 disk.md0p1.in‚ 10.97265625" proc.memory.rssMB‚ 3032.2265625 cpu‚25.178831178096747…)\\‘—$‚(:† ãçÔ‚£ü‡ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚104.494' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚26.1235 gpu.0.smClock‚1200 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%‰-¢‚˜$‚(:÷ ãçÔñ®¼ˆ disk.md0p1.out‚ 1037.12890625 network.recv‚ 3.344331e+06 disk.sda2.in‚ 5.56640625 disk.md0p1.in‚ 10.97265625" disk.sdb2.out‚1076.00537109375 disk.rbd2.in‚0" proc.memory.rssMB‚ 3032.2265625) proc.memory.percent‚18.50724220275879 network.sent‚ 2.1844695e+07% disk./.usageGB‚2975.3437309265137 disk.md0.in‚ 10.97265625$ memory_percent‚38.23566436767578" disk.nvme0n1.out‚ 57194.7890625 disk.dm-0.out‚121303.2734375 disk.sda1.in‚0 disk.md0.out‚ 1037.12890625 disk.rbd2.out‚ 2.74609375 disk.sdb.in‚5.40625 disk.sdb1.in‚0 disk.sda.in‚ 5.56640625) disk./.usagePercent‚83.19185599479975 disk.nvme0n1.in‚ 329972.65625 disk.sda1.out‚0" disk.nvme1n1.in‚201408.2421875! disk.sdb.out‚1076.00537109375' proc.memory.availableMB‚ 10119.46875" disk.sda2.out‚1076.00537109375 disk.dm-0.in‚531380.8984375! disk.nvme1n1.out‚ 64108.484375 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.sdb2.in‚5.40625 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 2.80859375! disk.sda.out‚1076.00537109375 cpu‚25.1786078820049680e å2™$- _wandbruntime‚9806  _runtime‚9806r%ZÊÉš$à ' lr-AdamW/pg1‚0.00021274469779289558 ' lr-AdamW/pg2‚0.00021274469779289558 trainer/global_step‚8399 % _timestamp‚1.7884737533941357e+09 _runtime‚9806.579112377 _step‚686Q_à Λ$È _step‚686 ' lr-AdamW/pg1‚0.00021274469779289558 ' lr-AdamW/pg2‚0.00021274469779289558 trainer/global_step‚8399 % _timestamp‚1.7884737533941357e+09 _runtime‚9806.579112377®¬ü¦:2œ$- _wandbruntime‚9806  _runtime‚9806‹\}î'$èO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95161314 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 159336885 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 228142598 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 159336885 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 228168942 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 % _timestamp‚1.7884737533953426e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚341385269401600 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 387479483 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚358063656803840 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16678387402240 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚358063908475392 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16678387402240 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚341385269401600 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚721 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚358063656803840 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚361639214665216 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 155647536 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 387505829 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚324930138864336 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 159336887 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282230784 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚341561256280770 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 159336887 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚378599813864448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚361639190725120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚341561502301214 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16631117416434 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚723 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16678621070848 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 228168942 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚687 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 228142598 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16960599199232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚341385287404544 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496,«\ˆ†ž$íO XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 387479483 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚358063656803840 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16678387402240 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚358063908475392 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16678387402240 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚341385269401600 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚721 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚358063656803840 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚361639214665216 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 155647536 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 387505829 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚324930138864336 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 159336887 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282230784 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚341561256280770 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 159336887 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚378599813864448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚361639190725120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 KEDeviceStatsMon¤`åmitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚341561502301214 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16631117416434 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚723 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16678621070848 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 228168942 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚687 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 228142598 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16960599199232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚341385287404544 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95161314 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 159336885 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 228142598 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 159336885 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 228168942 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 % _timestamp‚1.7884737533953426e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚341385269401600¯¸íŽ2Ÿ$- _wandbruntime‚9808  _runtime‚9808µø& $òM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 159355160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95172118 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚378643493020672 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚378643180571648 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 387550822 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 228195660 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚341600991147832 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚341600745127180 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚341424573441536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16680585110016 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚714 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 228195660 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚341424573441536 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16680585110016 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 159355160 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 228169290 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 155665777 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚361680898635264 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 trainer/global_step‚8399 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16680351428096 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚341424555438592 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16680351428096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚358105158551552 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚324967669221760 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16633075905420 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16962594385408 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16962305876480 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 387550822 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60493659 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _step‚688 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚95172116 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884737550157366e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚358105158551552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚341424555438592 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 387524450 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60492947 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 159355162 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚358104906866688 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _runtime‚9808.201610483 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155665063 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 159355162 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0=}Hgý&¡$÷M B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95172118 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚378643493020672 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚378643180571648 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 387550822 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 228195660 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚341600991147832 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚341600745127180 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚341424573441536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16680585110016 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚714 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 228195660 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚341424573441536 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16680585110016 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 159355160 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 228169290 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 155665777 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚361680898635264 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 trainer/global_step‚8399 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16680351428096 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚341424555438592 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16680351428096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚358105158551552 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚324967669221760 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16633075905420 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16962594385408 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16962305876480 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 387550822 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60493659 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _step‚688 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚95172116 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884737550157366e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚358105158551552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚341424555438592 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 387524450 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60492947 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 159355162 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚358104906866688 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _runtime‚9808.201610483 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155665063 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 159355162 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 159355160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚360126174360064 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚343349589254144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚343349571251200 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚380783679240704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16776603108864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 229480550 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 229506894 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 389731185 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301976576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚326806656735536 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 160250635 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 389757531 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚360126174360064 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚343349589254144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚343349571251200 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚380783679240704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16776603108864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 229480550 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 229506894 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 389731185 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301976576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚326806656735536 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 160250635 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 389757531 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16778567134720 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚343575201038888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚9870.456571452 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚343388875291136 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 229533612 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60841254 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 389802524 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚380827669777920 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚343388875291136 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16778567134720 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 160268912 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17062282034176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚360167676107776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 156553472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156552804 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 160268912 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 389802524 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 160268910 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚360167424422912 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 229533612 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 229507242 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚380827355231744 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60840588 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚363765097137664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚343388857288192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 389776152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚326844187092960 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16731242927220ù50Š_·$÷M K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 229507242 C9DeviceStóóçž&atsMonitor.on_train_batch_end/allocation.all.peak‚32249 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚380827355231744 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60840588 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚363765097137664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚343388857288192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 389776152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚326844187092960 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16731242927220 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 229507242 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95712218 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚666 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚363765073197568 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚343575447059540 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16778567134720 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚343575201038888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚9870.456571452 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚343388875291136 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 229533612 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60841254 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 389802524 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚380827669777920 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚343388875291136 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16778567134720 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 160268912 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17062282034176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚360167676107776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 156553472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156552804 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 160268912 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 389802524 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 160268910 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚360167424422912 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 229533612 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104´çܽË2¸$- _wandbruntime‚9870  _runtime‚9870Eµ4DY¹$Ó 0loss/FRt4/assignment_loss‚2.9681889802217483 /loss/FRt2/detection_loss‚1.6966679771835713 .loss/FBt3/detection_loss‚0.418021205370709 epoch‚6 1loss/FBt2/assignment_loss‚0.21440396738411494 .loss/FRt4/detection_loss‚3.349537527561188 1loss/SRqqt1/detection_loss‚1.9642152669953137 trainer/global_step‚8449 % _timestamp‚1.7884738172732344e+09 2loss/SRqqt3/assignment_loss‚0.9302329024723308 .loss/FBt1/detection_loss‚2.773778736591339 0loss/FRt3/assignment_loss‚0.8993036080347864 1loss/FBt3/assignment_loss‚0.16560418939927574 0loss/SRqqt4/detection_loss‚2.289389407634735 /loss/FBt4/detection_loss‚2.3504948019981384 .loss/FRt1/detection_loss‚2.694155177101493 /loss/FRt1/assignment_loss‚2.512947829440236 0loss/FRt2/assignment_loss‚3.1322878184525864 2loss/SRqqt1/assignment_loss‚1.7022350967475555 1loss/FBt1/assignment_loss‚0.03644334407211621 .loss/FRt3/detection_loss‚1.411274900267783 1loss/SRqqt2/detection_loss‚2.5140128674045688 1loss/SRqqt3/detection_loss‚0.7631116233427416 2loss/SRqqt4/assignment_loss‚1.5885584701318294 &loss/total_loss‚39.963335265867144 0loss/SRqqt2/assignment_loss‚1.32211332984509 0loss/FBt4/assignment_loss‚0.2012248235322193 .loss/FBt2/detection_loss‚2.065131414681673 _runtime‚9870.457964109 _step‚693ùQ^º$Ø 2loss/SRqqt3/assignment_loss‚0.9302329024723308 .loss/FBt1/detection_loss‚2.773778736591339 0loss/FRt3/assignment_loss‚0.8993036080347864 1loss/FBt3/assignment_loss‚0.16560418939927574 0loss/SRqqt4/detection_loss‚2.289389407634735 /loss/FBt4/detection_loss‚2.3504948019981384 .loss/FRt1/detection_loss‚2.694155177101493 /loss/FRt1/assignment_loss‚2.512947829440236 0loss/FRt2/assignment_loss‚3.1322878184525864 2loss/SRqqt1/assignment_loss‚1.7022350967475555 1loss/FBt1/assignment_loss‚0.03644334407211621 .loss/FRt3/detection_loss‚1.411274900267783 1loss/SRqqt2/detection_loss‚2.5140128674045688 1loss/SRqqt3/detection_loss‚0.7631116233427416 2loss/SRqqt4/assignment_loss‚1.5885584701318294 &loss/total_loss‚39.963335265867144 0loss/SRqqt2/assignment_loss‚1.32211332984509 0loss/FBt4/assignment_loss‚0.2012248235322193 .loss/FBt2/detection_loss‚2.065131414681673 _runtime‚9870.457964109 _step‚693 0loss/FRt4/assignment_loss‚2.9681889802217483 /loss/FRt2/detection_loss‚1.6966679771835713 .loss/FBt3/detection_loss‚0.418021205370709 epoch‚6 1loss/FBt2/assignment_loss‚0.21440396738411494 .loss/FRt4/detection_loss‚3.349537527561188 1loss/SRqqt1/detection_loss‚1.9642152669953137 trainer/global_step‚8449 % _timestamp‚1.7884738172732344e+09µ¾/wp»$‚(:„ çãçÔîŽî‡ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚32.0 gpu.0.powerWatts‚96.416' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚24.104 gpu.0.smClock‚1215 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0y¡}u¼$‚(:ê çãçÔèàĉ network.recv‚ 3.375075e+06 disk.nvme0n1.out‚ 57960.78125 disk.sda1.in‚0 disk.sdb.out‚1086.232421875% memory_percent‚38.234806060791016( proc.memory.availableMB‚ 10119.609375 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 2.80859375" disk.nvme1n1.in‚201408.2421875 disk.nvme0n1.in‚ 329972.65625 disk.sdb2.in‚5.40625 disk.sdb2.out‚1086.232421875 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.sda.out‚1086.232421875" proc.memory.rssMB‚ 3032.2265625) disk./.usagePercent‚83.19188617973033 disk.dm-0.in‚531380.8984375 disk.sda2.out‚1086.232421875 disk.md0.in‚ 10.97265625 disk.rbd2.out‚ 2.74609375 disk.sdb.in‚5.40625# disk./.usageGB‚2975.34481048584 disk.sda1.out‚0 disk.md0p1.in‚ 10.97265625 disk.md0.out‚ 1046.96875# disk.nvme1n1.out‚64214.63671875 disk.sdb1.in‚0) proc.memory.percent‚18.50724220275879 network.sent‚ 2.2084439e+07! disk.dm-0.out‚122175.41796875 disk.sda.in‚ 5.56640625 cpu‚25.177283982166276 disk.sda2.in‚ 5.56640625 disk.md0p1.out‚ 1046.96875 proc.cpu.threads‚17¹K8üŽ½$‚(:ƒ öãçÔ¤š‡ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚114.74' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.685 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0B@̃¾$‚(:ø öãçԚꈈ disk.nvme0n1.out‚ 58058.40625$ disk./.usageGB‚2975.343849182129 disk.rbd0.out‚ 2.80859375" disk.nvme1n1.out‚ 64214.8046875 disk.sdb.in‚5.40625 disk.nvme0n1.in‚ 329972.65625 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 2.74609375) proc.memory.percent‚18.50724220275879 cpu‚25.181478783956926 disk.sda2.in‚ 5.56640625" disk.sdb2.out‚1087.91064453125 disk.md0p1.out‚ 1048.58203125 disk.sdb2.in‚5.40625% memory_percent‚38.236284255981445 disk.md0p1.in‚ 10.97265625 disk.md0.in‚ 10.97265625 disk.sdb1.in‚0 disk.dm-0.in‚531380.8984375 disk.dm-0.out‚122273.2109375) proc.memory.availableMB‚ 10119.3671875 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.sda.in‚ 5.56640625 network.sent‚ 2.2171287e+07" disk.sda2.out‚1087.91064453125 disk.md0.out‚ 1048.58203125 disk.rbd0.in‚0.015625" disk.nvme1n1.in‚201408.2421875 network.recv‚ 3.382232e+06) disk./.usagePercent‚83.19185930127624 disk.sda1.in‚0! disk.sdb.out‚1087.91064453125! disk.sda.out‚1087.91064453125" proc.memory.rssMB‚ 3032.2265625¼NBº¿$‚(:… …äçÔ Žæ‡ gpu.0.gpu‚40.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚143.872' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.968 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—DX¥‚À$‚(:÷ …äçÔí–¨ˆ cpu‚25.186883149559304 disk.md0.out‚ 1049.87109375) proc.memory.percent‚18.50724220275879) disk./.usagePercent‚83.18989856071425 disk.md0p1.out‚ 1049.87109375 disk.md0.in‚ 10.97265625 disk.rbd2.in‚0 network.recv‚ 3.385989e+06 disk.nvme0n1.in‚ 329972.65625 disk.sdb2.in‚5.40625" proc.memory.rssMB‚ 3032.2265625 proc.cpu.threads‚17 network.sent‚ 2.2174759e+07$ disk./.usageGB‚2975.273723602295 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sda1.out‚0" disk.sda2.out‚1089.26611328125 disk.md0p1.in‚ 10.97265625 disk.rbd0.in‚0.015625 disk.dm-0.out‚122500.2890625" disk.nvme1n1.out‚ 64214.9140625 disk.sdb1.in‚0 disk.nvme0n1.out‚ 58295.125% memory_percent‚38.234829902648926 disk.rbd2.out‚ 2.74609375! disk.sdb.out‚1089.26611328125! disk.sda.out‚1089.26611328125 disk.dm-0.in‚531380.8984375 disk.sda2.in‚ 5.56640625 disk.rbd0.out‚ 2.80859375* proc.memory.availableMB‚10119.60546875 disk.sdb.in‚5.40625 disk.sda.in‚ 5.56640625" disk.sdb2.out‚1089.26611328125" disk.nvme1n1.in‚201408.2421875žX6‘Á$‚(:† ”äçԵ甇 gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­…Ô+Â$‚(:ö ”äçÔõé·ˆ disk.rbd0.out‚ 2.80859375 disk.sda.in‚ 5.56640625 disk.dm-0.out‚ 122688.171875" disk.nvme1n1.out‚ 64215.1328125 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sda2.in‚ 5.56640625" disk.nvme1n1.in‚201408.2421875) proc.memory.percent‚18.50724220275879) disk./.usagePercent‚83.17710622979423 disk.nvme0n1.in‚ 329972.65625" disk.sdb2.out‚1091.22119140625 disk.rbd2.in‚0% memory_percent‚38.236212730407715* proc.memory.availableMB‚10119.37890625$ disk./.usageGB‚2974.816207885742 disk.sdb1.in‚0 disk.dm-0.in‚531380.8984375! disk.sdb.out‚1091.22119140625! disk.sda.out‚1091.22119140625" disk.nvme0n1.out‚ 58473.0390625 disk.md0.in‚ 10.97265625 disk.md0.out‚ 1051.765625 disk.sdb.in‚5.40625" proc.memory.rssMB‚ 3032.2265625 disk.sdb2.in‚5.40625 disk.rbd2.out‚ 2.74609375 proc.cpu.threads‚17 disk.sda1.out‚0" disk.sda2.out‚1091.22119140625 disk.md0p1.in‚ 10.97265625 disk.md0p1.out‚ 1051.765625 disk.rbd0.in‚0.015625 cpu‚25.192146447027717 network.sent‚ 2.2182351e+07 network.recv‚ 3.390382e+06µÁ<’Ã$‚(:‡ £äçÔۚ‡ gpu.0.gpu‚39.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚142.313' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.57825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ú¨ ¤xÄ$‚(:í £äçÔ¥Ž„ˆ disk.sda.out‚1093.0009765625) proc.memory.availableMB‚ 10119.3828125) proc.memory.percent‚18.50724220275879) disk./.usagePercent‚83.17782842825723 disk.sdb.out‚1093.0009765625 disk.sdb1.out‚0 network.recv‚ 3.394168e+06! disk.nvme0n1.out‚ 58545.609375 disk.md0p1.in‚ 10.97265625 disk.nvme1n1.out‚ 64215.28125 cpu‚25.197446842753813% disk./.usageGB‚2974.8420372009277 disk.dm-0.in‚531380.8984375 disk.sdb.in‚5.40625 proc.cpu.threads‚17 disk.md0p1.out‚ 1053.46875 network.sent‚ 2.2187727e+07 disk.nvme0n1.in‚ 329972.65625" disk.nvme1n1.in‚201408.2421875% memory_percent‚38.236188888549805 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0.in‚ 10.97265625 disk.md0.out‚ 1053.46875 disk.rbd2.out‚ 2.74609375 disk.sdb1.in‚0 disk.sda.in‚ 5.56640625" proc.memory.rssMB‚ 3032.2265625 disk.dm-0.out‚ 122760.890625! disk.sda2.out‚1093.0009765625! disk.sdb2.out‚1093.0009765625 disk.rbd2.in‚0 disk.sda2.in‚ 5.56640625 disk.sdb2.in‚5.40625 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 2.80859375œ\42Å$- _wandbruntime‚9950  _runtime‚9950´»xU*Æ$ÏT [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 _step‚694 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚345372793806848 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188859832 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 230295898 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚16827568045056 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚362200379854848 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 391556101 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 230316609 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 230316609 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 193815040 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚345372793806848 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3532 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚61208958 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚365794714674688 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚17111811496448 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚328584504571440 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 391576814 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 565025792 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚61209344 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚17111658055680 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 560288777 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚16779686743806 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚16827392232960 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6966900736 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1300 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 177380864 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚16779858564278 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚582 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚273 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚273 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚345372811809792 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 391556101 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 565025792 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚328584521610800 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚273 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚16827568045056 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 161260203 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 157634406 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚96425062 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚96425060 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193815040 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20711 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 572522496 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 157634018 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 7134511104 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 391576814 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚382906526171136 % _timestamp‚1.7884738975048852e+09 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3532 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚345372811809792 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚582 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚8462 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 230295898 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚365794690734592 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚345364380175078 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚345364191315246 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20713 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚388 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 153440768 _runtime‚9950.690539401 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 161260203 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚386 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚309 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚362200186039808 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 161260205 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1319 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚01T²8Ç$ÔT FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 565025792 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚328584521610800 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚273 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚16827568045056 VHDeviceStatsMonitor.on_validationßiNÔ"%_batch_start/allocation.large_pool.freed‚ 161260203 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 157634406 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚96425062 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚96425060 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193815040 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20711 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 572522496 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 157634018 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 7134511104 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 391576814 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚382906526171136 % _timestamp‚1.7884738975048852e+09 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3532 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚345372811809792 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚582 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚8462 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 230295898 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚365794690734592 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚345364380175078 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚345364191315246 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20713 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚388 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 153440768 _runtime‚9950.690539401 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 161260203 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚386 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚309 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚362200186039808 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 161260205 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1319 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 _step‚694 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚345372793806848 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188859832 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 230295898 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚16827568045056 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚362200379854848 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 391556101 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 230316609 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 230316609 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 193815040 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚345372793806848 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3532 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚61208958 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚365794714674688 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚17111811496448 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚328584504571440 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 391576814 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 565025792 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚61209344 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚17111658055680 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 560288777 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚16779686743806 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚16827392232960 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6966900736 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1300 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 177380864 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚16779858564278 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚582 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚273 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚273 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚345372811809792 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 391556101 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚362230919389184 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚382937184403968 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1300 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 572522496 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 153440768 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚16780158951618 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚582 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 391601104 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚96436744 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚365825076428288 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6966900736 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 7134511104 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 153440768 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚16780158951618 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚582 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 391601104 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚96436744 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚365825076428288 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6966900736 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 7134511104 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚362230919389184 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚382937184403968 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1300 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 572522496 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 B&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5379585027694702 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9678362607955933 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5390145182609558 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.04940476268529892 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151560306549 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 _step‚696 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5331036448478699 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7004193067550659 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1468254029750824 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.08053861558437347 6 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16253206133842468 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 particle/accuracy_1_of_7‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.469933420419693 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4978187084197998 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5271317958831787 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.16959063708782196 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2826327383518219 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_7_of_7‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6447395086288452 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22564059495925903 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9820402264595032 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07941742241382599 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6085012555122375 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt1SRqqt0FBt/event_purity‚0.002910052891820669 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.38377973437309265 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1873534619808197 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9908550381660461 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16203755140304565 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 particle/accuracy_2_of_3‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 *jet/accuracy_2_of_6‚0.6602563858032227 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.44117647409439087 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.514162540435791 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9880177974700928 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23292413353919983 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9670635461807251 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 +"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.25 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9739024043083191 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 (particle/accuracy‚0.8543269038200378 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.8106060028076172 +jet/accuracy_2_of_3‚0.26691409945487976 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.27407410740852356 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9963507652282715 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 &particle/f_score‚0.420762300491333 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.51925128698349 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.35185182094573975 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013504411093890667 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.010758377611637115 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5411751866340637 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.19109195470809937 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5175347924232483 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 _runtime‚9968.419800122 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5268431901931763 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.28125 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13599875569343567 7"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.53115314245224 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3709373474121094 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.25057467818260193 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009770826436579227 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.21171170473098755 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.27500003576278687 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2161855846643448 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.7017543911933899 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.2051166146993637 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5595523715019226 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.392160028219223 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 +jet/accuracy_3_of_3‚0.02557099238038063 jet/accuracy_6_of_7‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5832262635231018 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 )jet/accuracy_2_of_4‚0.416019082069397 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5462963581085205 jet/accuracy_4_of_6‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5526955723762512 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3928605616092682 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.589516818523407 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.019607843831181526 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1717911660671234 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9496775269508362 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.6003085970878601 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5089327692985535 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.23305054008960724 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 particle/accuracy_2_of_6‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5396310091018677 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5734788179397583 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4895833730697632 jet/accuracy_3_of_7‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5214689373970032 7"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.54201340675354 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5357142686843872 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1748218685388565 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1315789520740509 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02304147742688656 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08502700179815292 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1481977254152298 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6363095641136169 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 *jet/accuracy_1_of_6‚0.9807692170143127 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5900792479515076 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04043741896748543 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8890267014503479 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4940776824951172 *jet/accuracy_1_of_5‚0.9684203863143921 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_7‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1960783749818802 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6230769753456116 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3923751711845398 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5878716707229614 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4322310984134674 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5379585027694702 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.19155606627464294 +jet/accuracy_3_of_5‚0.12661290168762207 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4805704355239868 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_7‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0069444444961845875 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_4_of_4‚0.010398948565125465 particle/accuracy_2_of_2‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1491018533706665 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.43013396859169006 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 )loss/val_total_loss‚40.98040613304041 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.03488372266292572 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 6 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.975907027721405 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 epoch‚6 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17294377088546753 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9811320900917053 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3902777433395386 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005777636077255011 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.15255099534988403 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 *"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17566144466400146 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.46530213952064514 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.05263157933950424 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9910197854042053 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03440878540277481 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_4_of_7‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 ,jet/accuracy_3_of_6‚0.057692307978868484 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.027021920308470726 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07875827699899673 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.680555522441864 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5614394545555115 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.15279729664325714 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9864019155502319 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 <"Purity/2FRt1SRqqt1FBt/event_purity‚0.0059523810632526875 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22908061742782593 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8224920630455017 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.4375 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.39259251952171326 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4358713626861572 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6427754759788513 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.11316872388124466 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14814293384552002 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9877446293830872 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5471420288085938 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2695518434047699 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3925236165523529 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5791262984275818 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0416666679084301 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20673269033432007 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,particle/sensitivity‚0.33906951546669006 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4583333432674408 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9900406002998352 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.306999146938324 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2261904776096344 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 8"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.463813841342926 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.11555555462837219 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ;"Purity/2FRt1SRqqt*FBt/event_purity‚0.004031242337077856 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2771376967430115 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3152259886264801 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0054946220479905605 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7validation_average_jet_accuracy‚0.45022398233413696 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.171875 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 particle/accuracy_5_of_6‚1 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.019607843831181526 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1628352403640747 particle/accuracy_1_of_5‚1 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.019771281629800797 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5800439119338989 particle/accuracy_3_of_4‚1 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1570248156785965 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.001792114693671465 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.9166666865348816 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6411341428756714 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.5134752988815308 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.021902700886130333 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.1080177053809166 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3360341787338257 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.05458220839500427 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17173832654953003 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 6"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4757080078125 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0416666679084301 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 particle/accuracy_1_of_2‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.21450616419315338 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.47769346833229065 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_7‚1 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.02298850566148758 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.036357514560222626 jet/accuracy_5_of_5‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9941977262496948 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1960928589105606 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735632538795471 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6112307906150818 *jet/accuracy_1_of_2‚0.7677459120750427 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9905831813812256 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0035757895093411207 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.43398791551589966 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4775862395763397 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.11395628750324249 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.553996741771698 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_3‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 8"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.550000011920929 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2913269102573395 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.3333333134651184 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_3_of_7‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9919098615646362 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9973641633987427 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9908677339553833 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4933333098888397 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5851190686225891 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20242814719676971 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5991379022598267 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9796274900436401 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.11555632203817368 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10193393379449844 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17566144466400146 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 particle/accuracy_1_of_6‚1 jet/accuracy_7_of_7‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5394089818000793 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5605393648147583 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.4736842215061188 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9898503422737122 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.28770509362220764 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.021579742431640625 particle/accuracy_4_of_6‚1 jet/accuracy_2_of_7‚0 % _timestamp‚1.7884739152314978e+09 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.547646701335907 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9331104159355164 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25938329100608826 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5967453122138977 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5769990682601929 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12568655610084534 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333432674408 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7954545617103577 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.21590907871723175 )jet/accuracy_2_of_5‚0.484639048576355 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6309027671813965 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1536378711462021 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5520833730697632 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9977453947067261 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/0FRt3SRqqt1FBtQZýäg/FBt_purity‚1 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41876041889190674 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_1‚0.5023359656333923 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 jet/accuracy_1_of_7‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9184396862983704 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5441092252731323 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.49478113651275635 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9928697347640991 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5428414344787598 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.12056738138198853 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4781630039215088 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9975420236587524 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43209874629974365 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.46786490082740784 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5821876525878906 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9035088419914246 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9910610318183899 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5783547163009644 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2841205298900604 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2857305109500885 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4722222089767456 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.47727271914482117 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.0946916937828064 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5555323362350464 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5864210724830627 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.18079093098640442 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1362183541059494 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4583333432674408 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 :"Purity/1FRt2SRqqt1FBt/event_purity‚0.01206896547228098 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15870241820812225 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5559867024421692 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15931223332881927 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5453745722770691 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 trainer/global_step‚8462 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 particle/accuracy_6_of_7‚1 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305Ý«_Š‹mÏ$ǧ 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13599875569343567 7"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.53115314245224 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3709373474121094 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.25057467818260193 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009770826436579227 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.21171170473098755 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.27500003576278687 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2161855846643448 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.7017543911933899 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.2051166146993637 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5595523715019226 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.392160028219223 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 +jet/accuracy_3_of_3‚0.02557099238038063 jet/accuracy_6_of_7‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5832262635231018 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 )jet/accuracy_2_of_4‚0.416019082069397 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5462963581085205 jet/accuracy_4_of_6‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5526955723762512 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3928605616092682 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.589516818523407 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.019607843831181526 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1717911660671234 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9496775269508362 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.6003085970878601 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5089327692985535 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.23305054008960724 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 particle/accuracy_2_of_6‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5396310091018677 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5734788179397583 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4895833730697632 jet/accuracy_3_of_7‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5214689373970032 7"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.54201340675354 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5357142686843872 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1748218685388565 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1315789520740509 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02304147742688656 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08502700179815292 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1481977254152298 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6363095641136169 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 *jet/accuracy_1_of_6‚0.9807692170143127 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5900792479515076 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04043741896748543 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8890267014503479 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4940776824951172 *jet/accuracy_1_of_5‚0.9684203863143921 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_7‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1960783749818802 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6230769753456116 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3923751711845398 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5878716707229614 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4322310984134674 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5379585027694702 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.19155606627464294 +jet/accuracy_3_of_5‚0.12661290168762207 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4805704355239868 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_7‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0069444444961845875 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_4_of_4‚0.010398948565125465 particle/accuracy_2_of_2‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1491018533706665 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.43013396859169006 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 )loss/val_total_loss‚40.98040613304041 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.03488372266292572 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 6 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.975907027721405 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 epoch‚6 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17294377088546753 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9811320900917053 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3902777433395386 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005777636077255011 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.15255099534988403 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 *"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17566144466400146 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.46530213952064514 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.05263157933950424 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9910197854042053 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03440878540277481 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_4_of_7‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 ,jet/accuracy_3_of_6‚0.057692307978868484 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.027021920308470726 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07875827699899673 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.680555522441864 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5614394545555115 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.15279729664325714 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9864019155502319 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 <"Purity/2FRt1SRqqt1FBt/event_purity‚0.0059523810632526875 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22908061742782593 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8224920630455017 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.4375 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.39259251952171326 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4358713626861572 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6427754759788513 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.11316872388124466 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14814293384552002 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9877446293830872 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5471420288085938 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2695518434047699 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3925236165523529 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5791262984275818 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0416666679084301 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20673269033432007 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,particle/sensitivity‚0.33906951546669006 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4583333432674408 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9900406002998352 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.306999146938324 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2261904776096344 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 8"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.463813841342926 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.11555555462837219 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ;"Purity/2FRt1SRqqt*FBt/event_purity‚0.004031242337077856 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2771376967430115 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3152259886264801 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0054946220479905605 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7validation_average_jet_accuracy‚0.45022398233413696 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.171875 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 particle/accuracy_5_of_6‚1 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.019607843831181526 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1628352403640747 particle/accuracy_1_of_5‚1 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.019771281629800797 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5800439119338989 particle/accuracy_3_of_4‚1 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1570248156785965 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.001792114693671465 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.9166666865348816 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6411341428756714 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.5134752988815308 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.021902700886130333 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.1080177053809166 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3360341787338257 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.05458220839500427 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17173832654953003 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 6"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4757080078125 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0416666679084301 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 particle/accuracy_1_of_2‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.21450616419315338 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.47769346833229065 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_7‚1 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.02298850566148758 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.036357514560222626 jet/accuracy_5_of_5‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9941977262496948 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1960928589105606 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735632538795471 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6112307906150818 *jet/accuracy_1_of_2‚0.7677459120750427 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9905831813812256 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0035757895093411207 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.43398791551589966 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4775862395763397 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.11395628750324249 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.553996741771698 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_3‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 8"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.550000011920929 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2913269102573395 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.3333333134651184 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_3_of_7‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9919098615646362 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9973641633987427 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9908677339553833 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4933333098888397 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5851190686225891 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20242814719676971 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5991379022598267 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9796274900436401 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.11555632203817368 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10193393379449844 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17566144466400146 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 particle/accuracy_1_of_6‚1 jet/accuracy_7_of_7‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5394089818000793 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5605393648147583 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.4736842215061188 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9898503422737122 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.28770509362220764 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.021579742431640625 particle/accuracy_4_of_6‚1 jet/accuracy_2_of_7‚0 % _timestamp‚1.7884739152314978e+09 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.547646701335907 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9331104159355164 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25938329100608826 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5967453122138977 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5769990682601929 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12568655610084534 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333432674408 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7954545617103577 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.21590907871723175 )jet/accuracy_2_of_5‚0.484639048576355 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6309027671813965 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1536378711462021 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5520833730697632 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9977453947067261 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41876041889190674 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_1‚0.5023359656333923 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 jet/accuracy_1_of_7‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9184396862983704 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5441092252731323 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.49478113651275635 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9928697347640991 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5428414344787598 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.12056738138198853 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4781630039215088 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9975420236587524 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43209874629974365 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.46786490082740784 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5821876525878906 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9035088419914246 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9910610318183899 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5783547163009644 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2841205298900604 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2857305109500885 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4722222089767456 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.47727271914482117 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.0946916937828064 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5555323362350464 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5864210724830627 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.18079093098640442 @&Purity/0FRt4SRqqt*FBt/eventç6‡ÒC&_proportion‚0.0004960317746736109 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1362183541059494 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4583333432674408 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 :"Purity/1FRt2SRqqt1FBt/event_purity‚0.01206896547228098 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15870241820812225 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5559867024421692 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15931223332881927 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5453745722770691 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 trainer/global_step‚8462 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 particle/accuracy_6_of_7‚1 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 particle/accuracy_6_of_6‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9815319180488586 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5379585027694702 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9678362607955933 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5390145182609558 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.04940476268529892 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151560306549 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 _step‚696 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5331036448478699 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7004193067550659 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1468254029750824 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.08053861558437347 6 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16253206133842468 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 particle/accuracy_1_of_7‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.469933420419693 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4978187084197998 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5271317958831787 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.16959063708782196 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2826327383518219 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_7_of_7‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6447395086288452 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22564059495925903 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9820402264595032 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07941742241382599 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6085012555122375 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt1SRqqt0FBt/event_purity‚0.002910052891820669 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.38377973437309265 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1873534619808197 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9908550381660461 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16203755140304565 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 particle/accuracy_2_of_3‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 *jet/accuracy_2_of_6‚0.6602563858032227 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.44117647409439087 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.514162540435791 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9880177974700928 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23292413353919983 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9670635461807251 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 +"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.25 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9739024043083191 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 (particle/accuracy‚0.8543269038200378 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.8106060028076172 +jet/accuracy_2_of_3‚0.26691409945487976 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.27407410740852356 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9963507652282715 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 &particle/f_score‚0.420762300491333 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.51925128698349 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.35185182094573975 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013504411093890667 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.010758377611637115 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5411751866340637 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.19109195470809937 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5175347924232483 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 _runtime‚9968.419800122 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5268431901931763 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.28125¸üKLYÐ$‚(:… ÁäçÔÀÇôˆ gpu.0.gpu‚36.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚103.544' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.886 gpu.0.smClock‚1200 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W–®ê€Ñ$‚(:õ ÁäçÔ»üÀŠ network.sent‚ 2.2298954e+07" disk.nvme1n1.in‚201408.2421875 disk.sdb1.in‚0 disk.sda.in‚ 5.56640625 disk.md0.out‚ 1056.36328125 disk.rbd0.out‚ 2.80859375$ memory_percent‚38.23838233947754) proc.memory.availableMB‚ 10119.0234375$ disk./.usageGB‚2975.095272064209 disk.md0p1.out‚ 1056.36328125 disk.sdb.in‚5.40625! disk.sdb2.out‚1096.0439453125 disk.rbd2.out‚ 2.74609375 disk.nvme0n1.in‚ 329972.65625 disk.sda2.in‚ 5.56640625" proc.memory.rssMB‚ 3032.2265625 proc.cpu.threads‚17! disk.sda2.out‚1096.0439453125 network.recv‚ 3.405731e+06 disk.sdb.out‚1096.0439453125 disk.sdb1.out‚0 disk.sda.out‚1096.0439453125# disk.nvme0n1.out‚58842.84765625 disk.dm-0.out‚123229.0234375 disk.sda1.in‚0( disk./.usagePercent‚83.1849089810216 disk.dm-0.in‚531380.8984375 disk.sdb2.in‚5.40625 disk.md0.in‚ 10.97265625 disk.sda1.out‚0) proc.memory.percent‚18.50724220275879 cpu‚25.20573452263165# disk.nvme1n1.out‚64386.17578125 disk.md0p1.in‚ 10.97265625 disk.rbd0.in‚0.015625 disk.rbd2.in‚0h¨kÒ$‚(:… ÐäçÔÇÖ‡ gpu.0.gpu‚14.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚32.0 gpu.0.powerWatts‚93.354' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚23.3385 gpu.0.smClock‚1215 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ù:Ó„Ó$‚(:ù ÐäçÔœž²ˆ network.recv‚ 3.413219e+06% disk./.usageGB‚2975.0946044921875 disk.sda.in‚ 5.56640625 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0p1.out‚ 1058.2578125" disk.sdb2.out‚1098.00244140625" disk.sda2.out‚1098.00244140625) disk./.usagePercent‚83.18489031542848 disk.sdb2.in‚5.40625 disk.md0.in‚ 10.97265625 disk.md0.out‚ 1058.2578125 disk.rbd2.out‚ 2.74609375 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 2.80859375! disk.sdb.out‚1098.00244140625 disk.dm-0.in‚531380.8984375" proc.memory.rssMB‚ 3031.2421875" disk.nvme1n1.in‚201408.2421875 disk.sdb.in‚5.40625 disk.sdb1.out‚0 disk.nvme0n1.in‚ 329972.65625# disk.nvme0n1.out‚59044.49609375 disk.sda2.in‚ 5.56640625$ memory_percent‚38.23118209838867( proc.memory.availableMB‚ 10120.203125 disk.rbd2.in‚0! disk.sda.out‚1098.00244140625! disk.dm-0.out‚123431.03515625 cpu‚25.205437788939427 proc.cpu.threads‚17" disk.nvme1n1.out‚ 64386.5390625 disk.md0p1.in‚ 10.97265625 network.sent‚ 2.2425012e+07 disk.sdb1.in‚0) proc.memory.percent‚18.50123405456543À3ÛÔ$‚(:… ßäçÔ§Šº‡ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚32.0 gpu.0.powerWatts‚100.482' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.1205 gpu.0.smClock‚1185 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:‚î…Õ$‚(:ú ßäçÔ ”׈# disk.nvme1n1.out‚64557.37109375 disk.dm-0.in‚531380.8984375 disk.sda1.in‚0 disk.md0p1.in‚ 10.97265625" disk.sdb2.out‚1099.36376953125 disk.md0.in‚ 10.97265625* proc.memory.availableMB‚10120.33203125 network.sent‚ 2.2428066e+07" proc.memory.rssMB‚ 3031.2421875) proc.memory.percent‚18.50123405456543$ disk./.usageGB‚2975.343925476074" disk.nvme1n1.in‚201408.2421875 disk.nvme0n1.in‚ 329972.65625 disk.dm-0.out‚123751.4453125 disk.md0.out‚ 1059.55859375 network.recv‚ 3.416945e+06 disk.rbd2.in‚0! disk.sdb.out‚1099.36376953125# disk.nvme0n1.out‚59194.07421875 disk.rbd0.in‚0.015625 disk.sda1.out‚0" disk.sda2.out‚1099.36376953125 disk.sdb2.in‚5.40625 disk.rbd2.out‚ 2.74609375 disk.rbd0.out‚ 2.80859375 disk.sdb.in‚5.40625 disk.sda2.in‚ 5.56640625 disk.md0p1.out‚ 1059.55859375 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda.in‚ 5.56640625! disk.sda.out‚1099.36376953125$ memory_percent‚38.23039531707764 cpu‚25.2051292437283 proc.cpu.threads‚17) disk./.usagePercent‚83.19186143448688P·Æ¹4Ö$/ _wandbruntime‚10018  _runtime‚10018v ·¢É×$à ' lr-AdamW/pg1‚0.00020241372643999198 ' lr-AdamW/pg2‚0.00020241372643999198 trainer/global_step‚8499 $ _timestamp‚1.788473965380087e+09 _runtime‚10018.565160237 _step‚697ÄëtyÎØ$È ' lr-AdamW/pg2‚0.00020241372643999198 trainer/global_step‚8499 $ _timestamp‚1.788473965380087e+09 _runtime‚10018.565160237 _step‚697 ' lr-AdamW/pg1‚0.00020241372643999198¹·ä¯E4Ù$/ _wandbruntime‚10018  _runtime‚10018|•î'Ú$èO H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16904922660864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 393541440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 162133453 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 231407987 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 231381643 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚347208028580864 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 231381643 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚690 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚698 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚367720378206720 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚330293432533552 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 162133451 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚367720402146816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚364112951241728 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚96977482 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17190507356672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚364112951241728 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16904922660864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16904688992256 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚347208028580864 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61536772 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 162133451 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚384910909503488 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚364112699570176 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚330293449572912 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 393515094 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚347150438250980 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16856988678068Ào)v<Û$íO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚347150192230536 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 $ _timestamp‚1.788473965381651e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚364112699570176 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚347208010577920 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 158513564 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚384910592846848 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚8499 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _runtime‚10018.566921795 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 162133453 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚347208010577920 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 162133451 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚384910909503488 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚364112699570176 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚330293449572912 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 393515094 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚347150438250980 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16856988678068 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16904922660864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 393541440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 162133453 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 231407987 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 231381643 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚347208028580864 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 231381643 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚690 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚698 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚367720378206720 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚330293432533552 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 162133451 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚367720402146816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚364112951241728 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚96977482 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17190507356672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚364112951241728 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16904922660864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.sµSΣ· mall_pool.freed‚16904688992256 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚347208028580864 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61536772 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158531004 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚347247296614912 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16906653018112 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚347247314617856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17192495694336 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 _runtime‚10019.778463736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 393586433 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16906886700032 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚384954265167872 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 231434705 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17192202991104 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚330330962890976 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16858718185970 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚347247314617856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚96988284 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚367762062176768 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚384954581811200 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16906886700032 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16858947167262 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚687 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 158531693 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚347247296614912 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 231434705 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚347189927097598 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚330330979930336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 $ _timestamp‚1.788473966592728e+09 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16906653018112 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚364153949633024 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚96988282 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 393560061 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 393586433 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61542722 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚364154201317888 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 162151726 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17192202991104 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚330330962890976 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16858718185970 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚347247314617856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚96988284 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚367762062176768 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚384954581811200 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16906886700032 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16858947167262 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚687 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 158531693 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚347247296614912 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 231434705 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚347189927097598 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚330330979930336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 $ _timestamp‚1.788473966592728e+09 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16906653018112 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚364153949633024 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚96988282 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 393560061 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 393586433 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61542722 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚364154201317888 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 162151726 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158531004 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚347247296614912 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16906653018112 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚347247314617856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17192495694336 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 _runtime‚10019.778463736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 393586433 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16906886700032 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚384954265167872»tg|4ß$/ _wandbruntime‚10019  _runtime‚10019©Å·^à$Ø 2loss/SRqqt2/assignment_loss‚1.6125277246038119 _runtime‚10019.782634515 0loss/FRt3/assignment_loss‚1.1661302803291216 2loss/SRqqt1/assignment_loss‚1.7007304486451726 1loss/SRqqt4/assignment_loss‚1.451128362762657 1loss/FBt1/assignment_loss‚0.29818256079828676 /loss/FRt1/detection_loss‚2.8724937697251636 0loss/SRqqt1/detection_loss‚2.206245052092003 1loss/SRqqt3/detection_loss‚0.7600879283055015 -loss/FBt1/detection_loss‚2.50451300235895 1loss/FBt3/assignment_loss‚0.11401009239257102 1loss/SRqqt2/detection_loss‚2.4207257936398188 epoch‚7 0loss/FRt2/assignment_loss‚3.3848109931656807 /loss/FRt2/detection_loss‚2.3570972498714444 0loss/FRt1/assignment_loss‚2.3444604516029357 0loss/SRqqt3/assignment_loss‚0.81773547426514 /loss/FRt3/detection_loss‚1.4698941187622647 /loss/FRt4/detection_loss‚3.2119367433630903 1loss/SRqqt4/detection_loss‚2.8810582716356623 /loss/FBt2/detection_loss‚1.7200369170078864 1loss/FBt2/assignment_loss‚0.17846769392308096 1loss/FBt4/assignment_loss‚0.34223763046267264 %loss/total_loss‚41.23100037194306 % _timestamp‚1.7884739665959973e+09 /loss/FBt3/detection_loss‚0.3330547795514576 /loss/FBt4/detection_loss‚2.4564029574394226 0loss/FRt4/assignment_loss‚2.6270320752392644 trainer/global_step‚8499 _step‚700§2 cá$Ý 0loss/SRqqt3/assignment_loss‚0.81773547426514 /loss/FRt3/detection_loss‚1.4698941187622647 /loss/FRt4/detection_loss‚3.2119367433630903 1loss/SRqqt4/detection_loss‚2.8810582716356623 /loss/FBt2/detection_loss‚1.7200369170078864 1loss/FBt2/assignment_loss‚0.17846769392308096 1loss/FBt4/assignment_loss‚0.34223763046267264 %loss/total_loss‚41.23100037194306 % _timestamp‚1.7884739665959973e+09 /loss/FBt3/detection_loss‚0.3330547795514576 /loss/FBt4/detection_loss‚2.4564029574394226 0loss/FRt4/assignment_loss‚2.6270320752392644 trainer/global_step‚8499 _step‚700 2loss/SRqqt2/assignment_loss‚1.6125277246038119 _runtime‚10019.782634515 0loss/FRt3/assignment_loss‚1.1661302803291216 2loss/SRqqt1/assignment_loss‚1.7007304486451726 1loss/SRqqt4/assignment_loss‚1.451128362762657 1loss/FBt1/assignment_loss‚0.29818256079828676 /loss/FRt1/detection_loss‚2.8724937697251636 0loss/SRqqt1/detection_loss‚2.206245052092003 1loss/SRqqt3/detection_loss‚0.7600879283055015 -loss/FBt1/detection_loss‚2.50451300235895 1loss/FBt3/assignment_loss‚0.11401009239257102 1loss/SRqqt2/detection_loss‚2.4207257936398188 epoch‚7 0loss/FRt2/assignment_loss‚3.3848109931656807 /loss/FRt2/detection_loss‚2.3570972498714444 0loss/FRt1/assignment_loss‚2.3444604516029357¼óÔb®’â$‚(:‡ îäçÔÄÇ‘‡ gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚103.267' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.81675 gpu.0.smClock‚1185 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[3¥‚ã$‚(:÷ îäçÔÞÛˆ" disk.sdb2.out‚1100.99462890625 disk.sda.in‚ 5.56640625 disk.md0p1.in‚ 10.97265625 disk.md0p1.out‚ 1061.10546875 disk.rbd0.in‚0.015625# disk.nvme1n1.out‚64557.82421875! disk.sdb.out‚1100.99462890625# memory_percent‚38.2401704788208# disk./.usageGB‚2975.34379196167 disk.rbd0.out‚ 2.80859375 disk.sdb1.in‚0! disk.sda.out‚1100.99462890625 disk.sda2.in‚ 5.56640625 disk.sdb2.in‚5.40625 disk.md0.out‚ 1061.10546875" proc.memory.rssMB‚ 3031.2421875 disk.rbd2.out‚ 2.74609375" disk.nvme1n1.in‚201408.2421875 disk.nvme0n1.in‚ 329972.65625 cpu‚25.204798157099365 disk.dm-0.out‚ 123979.03125) disk./.usagePercent‚83.19185770136825 disk.rbd2.in‚0 disk.sdb1.out‚0) proc.memory.percent‚18.50123405456543 network.sent‚ 2.244735e+07 network.recv‚ 3.421522e+06 disk.md0.in‚ 10.97265625# disk.nvme0n1.out‚59421.20703125 disk.sda1.in‚0" disk.sda2.out‚1100.99462890625 proc.cpu.threads‚17 disk.sdb.in‚5.40625 disk.dm-0.in‚531380.8984375 disk.sda1.out‚0* proc.memory.availableMB‚10118.73046875‚dð/‘ä$‚(:† ýäçÔÃÿʇ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚89.619' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚22.40475 gpu.0.smClock‚1200 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0r!V~å$‚(:ó ýäçÔ²´ˆ! disk.dm-0.out‚124031.28515625 disk.sda1.out‚0 network.sent‚ 2.2544354e+07" disk.nvme1n1.in‚201408.2421875 disk.sda.out‚1103.0869140625# memory_percent‚38.2411003112793 proc.cpu.threads‚17 disk.rbd2.out‚ 2.74609375 disk.sda1.in‚0 disk.md0p1.in‚ 10.97265625) proc.memory.percent‚18.50123405456543 cpu‚25.204544280320462 disk.sdb1.in‚0 disk.sdb1.out‚0! disk.sda2.out‚1103.0869140625! disk.sdb2.out‚1103.0869140625 disk.md0.out‚ 1063.1484375 disk.rbd2.in‚0 disk.nvme0n1.in‚ 329972.65625( proc.memory.availableMB‚ 10118.578125) disk./.usagePercent‚83.18989856071425 disk.sdb2.in‚5.40625 disk.md0.in‚ 10.97265625 disk.rbd0.in‚0.015625 disk.sda.in‚ 5.56640625# disk.nvme0n1.out‚59473.33984375 disk.sda2.in‚ 5.56640625 disk.md0p1.out‚ 1063.1484375" proc.memory.rssMB‚ 3031.2421875 network.recv‚ 3.428549e+06$ disk./.usageGB‚2975.273723602295 disk.rbd0.out‚ 2.80859375" disk.nvme1n1.out‚ 64557.9453125 disk.sdb.in‚5.40625 disk.sdb.out‚1103.0869140625 disk.dm-0.in‚531380.8984375^Iúþ‘æ$‚(:† ŒåçÔýúÚ‡ gpu.0.gpu‚38.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚104.494' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚26.1235 gpu.0.smClock‚1215 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NWïç$‚(:ô ŒåçÔÛ똈 disk.sda.out‚1104.7412109375 disk.sda1.out‚0 disk.sda2.in‚ 5.56640625 disk.rbd0.out‚ 2.80859375 proc.cpu.threads‚17% disk./.usageGB‚2974.8162231445312 disk.md0p1.in‚ 10.97265625 disk.sdb2.in‚5.40625 disk.sda.in‚ 5.56640625! disk.sda2.out‚1104.7412109375) proc.memory.availableMB‚ 10118.8203125 network.recv‚ 3.432923e+06 disk.rbd2.in‚0 disk.dm-0.out‚ 124147.15625 disk.sdb.out‚1104.7412109375 disk.sdb1.out‚0 disk.dm-0.in‚531380.8984375 disk.sda1.in‚0$ memory_percent‚38.23962211608887" proc.memory.rssMB‚ 3031.2421875! disk.sdb2.out‚1104.7412109375 disk.md0.out‚ 1064.74609375 disk.rbd2.out‚ 2.74609375" disk.nvme1n1.in‚201408.2421875" disk.nvme1n1.out‚ 64558.1328125" disk.nvme0n1.out‚ 59589.0234375) proc.memory.percent‚18.50123405456543 network.sent‚ 2.2550376e+07) disk./.usagePercent‚83.17710665643637 disk.md0p1.out‚ 1064.74609375 disk.md0.in‚ 10.97265625 disk.rbd0.in‚0.015625 disk.sdb.in‚5.40625 disk.sdb1.in‚0 disk.nvme0n1.in‚ 329972.65625 cpu‚25.204265914674618Ã|:’è$‚(:‡ ›åçÔÏ¥Ô‡ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚103.885' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.97125 gpu.0.smClock‚1215 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ênE‚ué$‚(:ê ›åçÔÏÇ·ˆ) disk./.usagePercent‚83.17782896155988 disk.md0.in‚ 10.97265625 disk.rbd2.in‚0 disk.dm-0.in‚531380.8984375 disk.sda2.out‚ 1106.18359375" proc.memory.rssMB‚ 3031.2421875 disk.sda2.in‚ 5.56640625 disk.sda.in‚ 5.56640625 disk.sda.out‚ 1106.18359375 disk.dm-0.out‚ 124219.875 disk.sda1.in‚0 disk.md0.out‚ 1066.13671875 disk.sdb1.in‚0$ memory_percent‚38.23959827423096 disk.rbd2.out‚ 2.74609375 disk.sdb1.out‚0) proc.memory.percent‚18.50123405456543 cpu‚25.203985993396024 network.recv‚ 3.436614e+06 disk.sda1.out‚0 disk.md0p1.out‚ 1066.13671875 disk.sdb2.in‚5.40625 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 2.80859375 disk.sdb.in‚5.40625 network.sent‚ 2.2553472e+07" disk.nvme1n1.out‚ 64558.3046875* proc.memory.availableMB‚10118.82421875 proc.cpu.threads‚17" disk.nvme1n1.in‚201408.2421875 disk.sdb.out‚ 1106.18359375$ disk./.usageGB‚2974.842056274414 disk.sdb2.out‚ 1106.18359375 disk.nvme0n1.in‚ 329972.65625" disk.nvme0n1.out‚ 59662.6953125 disk.md0p1.in‚ 10.97265625iB×iŽê$‚(:ƒ ªåçÔÍ߶‡ gpu.0.gpu‚38.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚105.72' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚26.43 gpu.0.smClock‚1230 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðÕ2Jìë$‚(:ö ªåçÔ󶦈 disk.rbd0.out‚ 2.80859375 disk.sdb1.in‚0 disk.md0p1.in‚ 10.97265625$ memory_percent‚38.23976516723633 disk.rbd2.out‚ 2.74609375" disk.nvme1n1.in‚201408.2421875 disk.sdb.out‚1107.7021484375 disk.sda.out‚1107.7021484375! disk.dm-0.out‚124438.27734375 disk.sda1.out‚0( proc.memory.availableMB‚ 10118.796875 proc.cpu.threads‚17) disk./.usagePercent‚83.17787941199158 disk.sdb2.in‚5.40625 disk.md0.in‚ 10.9Éi&¹•7265625 disk.sda2.in‚ 5.56640625 cpu‚25.20370890902327 network.recv‚ 3.441185e+06 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.sda.in‚ 5.56640625! disk.sdb2.out‚1107.7021484375" disk.nvme0n1.out‚ 59805.6953125) proc.memory.percent‚18.50123405456543% disk./.usageGB‚2974.8438606262207 disk.nvme0n1.in‚ 329972.65625! disk.sda2.out‚1107.7021484375 disk.md0p1.out‚ 1067.58984375" proc.memory.rssMB‚ 3031.2421875 network.sent‚ 2.2559768e+07# disk.nvme1n1.out‚64632.58203125 disk.sda1.in‚0 disk.md0.out‚ 1067.58984375 disk.sdb.in‚5.40625 disk.dm-0.in‚531380.8984375*¯{4ì$/ _wandbruntime‚10080  _runtime‚10080 ‘nÉí$à ' lr-AdamW/pg1‚0.00019731329170073628 ' lr-AdamW/pg2‚0.00019731329170073628 trainer/global_step‚8549 % _timestamp‚1.7884740272271566e+09 _runtime‚10080.41223553 _step‚701õÚ´Îî$È % _timestamp‚1.7884740272271566e+09 _runtime‚10080.41223553 _step‚701 ' lr-AdamW/pg1‚0.00019731329170073628 ' lr-AdamW/pg2‚0.00019731329170073628 trainer/global_step‚8549½Þ½&þ4ï$/ _wandbruntime‚10080  _runtime‚10080¸zÕûî'ð$èO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 232719595 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17290025950208 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚695 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 159396596 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚332169950404752 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 _step‚702 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 163047201 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚366175468797952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚97517582 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17002904698880 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚369804576709120 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16954697737492 trainer/global_step‚8549 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 395766796 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 232719595 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17002904698880 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 163047203 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚693 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚332169967444112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚349172330430464 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚349172312427520 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16954926718576 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 163047201 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 232745939 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚366175217126400 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚366175217126400 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17003138367488 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 232745939 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17290025950208 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚695 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 159396596 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚332169950404752 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 _step‚702 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 163047201 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚366175468797952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚97517582 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17002904698880 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚369804576709120 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16954697737492 trainer/global_step‚8549 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 395766796 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 232719595 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17002904698880 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 163047203 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚693 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚332169967444112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚349172330430464 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚349172312427520 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16954926718576 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 163047201 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 232745939 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚366175217126400 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚366175217126400 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17003138367488 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 232745939 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚332207497801536 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17004868724736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 232772657 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17005102406656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 163065476 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 159414248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚349211616467456 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚332207480762176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 163065478 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚349164383009306 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚349164136988654 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 395811763 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 304060416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚8549 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚97528384 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 163065476 _step‚703 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚349211598464512 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚349211598464512 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 159414919 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61885866 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17292309066240 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16956656226478 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280120320 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚366216718874112 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚349211616467456 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 395811763 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚387138289625088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17005102406656 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚366216467189248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884740284073315e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17004868724736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16956885207770 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 395838135 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚671 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17292028945920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 232746287 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚669 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚366216718874112 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 232746287 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚387138593685504 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚369846260679168 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 163065478 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 395838135pÒƒô$øM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚671 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17292028945920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚š*4_æ"26372 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 232746287 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚669 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚366216718874112 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 232746287 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚387138593685504 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚369846260679168 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 163065478 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 395838135 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61886535 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚332207497801536 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17004868724736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 232772657 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17005102406656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 163065476 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 159414248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚349211616467456 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚332207480762176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 163065478 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚349164383009306 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚349164136988654 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 395811763 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 304060416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚8549 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚97528384 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 163065476 _step‚703 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚349211598464512 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚349211598464512 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 159414919 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61885866 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17292309066240 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16956656226478 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280120320 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚366216718874112 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚349211616467456 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 395811763 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚387138289625088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17005102406656 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚366216467189248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884740284073315e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17004868724736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16956885207770 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 395838135 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 160281414 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚368237986354176 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚368237734682624 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚351136614277120 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17101120405504 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62224409 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚10142.545068069 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17101354074112 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚389278957793280 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17389872216576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚351136614277120 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 234057547 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚351099104053952 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 398018498 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17052635778000 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 % _timestamp‚1.7884740893600507e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17101120405504 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62223734 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚351099350074396 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚368237734682624 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 234057547 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 398044844 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚334046468275952 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17101354074112 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17390158641664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104!>„…%îO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98057682 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 234083891 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚675 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚371888799151616 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚8599 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚677 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 160282091 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚351136632280064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚351136632280064 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚389278647428096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 160281414 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚368237986354176 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚368237734682624 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚351136614277120 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17101120405504 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62224409 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚10142.545068069 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17101354074112 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚389278957793280 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17389872216576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚351136614277120 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 234057547 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚351099104053952 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 398018498 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17052635778000 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 % _timestamp‚1.7884740893600507e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17101120405504 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62223734 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚351099350074396 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚368237734682624 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 234057547 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 398044844 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚334046468275952 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17101354074112 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17390158641664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0ÂÍy¢4†%/ _wandbruntime‚10143  _runtime‚10143×èÙDù&‡%óM F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚351175900314112 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _step‚707 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62230426 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 163979228 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚669 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚389322326114304 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚368278984745472 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 234084239 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 160298908 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚389322640660480 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17054823248278 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚667 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚351138592900362 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚351175918317056 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17103318113280 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17392157538816 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 398089837 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 398063465 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 398063465 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17103084431360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 163979228 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 398089837 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98068482 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17054594266986 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17391866932736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚351138838921014 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 234110609 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17103084431360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 trainer/global_step‚8599 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚334083998633376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98068484 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚334084015672736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚368279236430336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚371930459181568 % _timestamp‚1.7884740903019235e+09 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 160299577 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 163979226 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚351175900314112'>¶pþ&ˆ%øM UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚669 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚389322326114304 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚368278984745472 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 234084239 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 160298908 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚389322640660480 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17054823248278 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚667 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚351138592900362 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚351175918317056 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17103318113280 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17392157538816 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 398089837 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 398063465 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 398063465 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17103084431360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 163979228 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 398089837 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98068482 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17054594266986 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17391866932736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚351138838921014 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 234110609 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17103084431360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 trainer/global_step‚8599 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚334083998633376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98068484 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚334084015672736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚368279236430336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚371930459181568 % _timestamp‚1.7884740903019235e+09 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 160299577 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 163979226 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚351175900314112 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚351175900314112 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _step‚707 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62230426 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 163979228 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136Ã**Ëæ4‰%/ _wandbruntime‚10143  _runtime‚101438ër·ZŠ%Ô epoch‚7 /loss/FRt2/assignment_loss‚3.316341292858124 2loss/SRqqt3/assignment_loss‚1.0166123185046418 1loss/SRqqt4/detection_loss‚2.4525778227382236 /loss/FBt4/detection_loss‚2.3792736615453447 2loss/SRqqt2/assignment_loss‚1.8033223793926565 1loss/FBt2/assignment_loss‚0.18486466462115914 1loss/SRqqt3/detection_loss‚0.9144816928773477 .loss/FBt2/detection_loss‚2.122843772172928 trainer/global_step‚8599 1loss/SRqqt1/assignment_loss‚1.230130025293938 0loss/FBt1/assignment_loss‚0.0720073162082009 /loss/FRt3/detection_loss‚1.4952666072814322 0loss/SRqqt1/detection_loss‚1.858104653829752 _runtime‚10143.489145601 0loss/FBt4/assignment_loss‚0.3729458259684511 /loss/FRt2/detection_loss‚2.0805057155620306 _step‚708 /loss/FRt4/assignment_loss‚3.199422659295978 1loss/SRqqt2/detection_loss‚2.1546997680808557 0loss/FBt3/detection_loss‚0.40211107206220426 0loss/FRt1/assignment_loss‚3.0790728601542385 0loss/FBt3/assignment_loss‚0.0621985013108944 -loss/FRt4/detection_loss‚2.73323960015268 .loss/FBt1/detection_loss‚2.292813811983381 &loss/total_loss‚40.860215958258976 $ _timestamp‚1.788474090304453e+09 0loss/FRt3/assignment_loss‚1.1785233928196466 2loss/SRqqt4/assignment_loss‚1.5689778406586912 /loss/FRt1/detection_loss‚2.8898787028861768ó ÐP_‹%Ù /loss/FRt3/detection_loss‚1.4952666072814322 0loss/SRqqt1/detection_loss‚1.858104653829752 _runtime‚10143.489145601 0loss/FBt4/assignment_loss‚0.3729458259684511 /loss/FRt2/detection_loss‚2.0805057155620306 _step‚708 /loss/FRt4/assignment_loss‚3.199422659295978 1loss/SRqqt2/detection_loss‚2.1546997680808557 0loss/FBt3/detection_loss‚0.40211107206220426 0loss/FRt1/assignment_loss‚3.0790728601542385 0loss/FBt3/assignment_loss‚0.0621985013108944 -loss/FRt4/detection_loss‚2.73323960015268 .loss/FBt1/detection_loss‚2.292813811983381 &loss/total_loss‚40.860215958258976 $ _timestamp‚1.788474090304453e+09 0loss/FRt3/assignment_loss‚1.1785233928196466 2loss/SRqqt4/assignment_loss‚1.5689778406586912 /loss/FRt1/detection_loss‚2.8898787028861768 epoch‚7 /loss/FRt2/assignment_loss‚3.316341292858124 2loss/SRqqt3/assignment_loss‚1.0166123185046418 1loss/SRqqt4/detection_loss‚2.4525778227382236 /loss/FBt4/detection_loss‚2.3792736615453447 2loss/SRqqt2/assignment_loss‚1.8033223793926565 1loss/FBt2/assignment_loss‚0.18486466462115914 1loss/SRqqt3/detection_loss‚0.9144816928773477 .loss/FBt2/detection_loss‚2.122843772172928 trainer/global_step‚8599 1loss/SRqqt1/assignment_loss‚1.230130025293938 0loss/FBt1/assignment_loss‚0.0720073162082009Ä6*D‘Œ%‚(:† õåçÔÝëú† gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚122.486' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.6215 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€ø±†%‚(:û õåçÔöµÃ‡ disk.sda1.out‚0) proc.memory.availableMB‚ 10121.1328125) proc.memory.percent‚18.50123405456543! disk.nvme1n1.out‚ 65141.671875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.dm-0.in‚531380.8984375 network.recv‚ 3.465196e+06 disk.nvme0n1.in‚ 329972.65625" proc.memory.rssMB‚ 3031.2421875 proc.cpu.threads‚17 disk.rbd0.in‚0.015625" disk.nvme1n1.in‚201408.2421875# disk.nvme0n1.out‚60094.26953125 disk.sda1.in‚0 disk.sda2.in‚ 5.56640625 disk.md0p1.in‚ 10.97265625) disk./.usagePercent‚83.17710974959179 disk.rbd0.out‚ 2.80859375 disk.sdb.in‚5.40625! disk.sda.out‚1116.23291015625 network.sent‚ 2.2691124e+07" disk.sdb2.out‚1116.23291015625 disk.rbd2.out‚ 2.74609375! disk.sdb.out‚1116.23291015625 disk.md0p1.out‚ 1075.79296875 cpu‚25.202385092542617$ disk./.usageGB‚2974.816333770752 disk.sda.in‚ 5.56640625 disk.sdb2.in‚5.40625 disk.rbd2.in‚0% memory_percent‚38.225507736206055" disk.sda2.out‚1116.23291015625 disk.md0.in‚ 10.97265625 disk.md0.out‚ 1075.79296875! disk.dm-0.out‚125235.94140625°¹g‘Ž%‚(:† „æçÔß‹ð… gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚102.658' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.6645 gpu.0.smClock‚1170 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÄH4 o%‚(:ä „æçÔêϦ‡ disk.rbd0.out‚ 2.80859375 disk.rbd2.out‚ 2.74609375 disk.sdb.in‚5.40625 disk.sda1.in‚0 disk.md0p1.out‚ 1077.296875) proc.memory.percent‚18.50123405456543 proc.cpu.threads‚17 disk.rbd2.in‚0" disk.nvme1n1.in‚201408.2421875 disk.sdb1.out‚0 disk.sdb2.out‚ 1117.80859375* proc.memory.availableMB‚10120.73046875 cpu‚25.202113241276066) disk./.usagePercent‚83.17711017623391 disk.sda.in‚ 5.56640625 disk.sda2.in‚ 5.56640625 disk.sdb1.in‚0 disk.dm-0.in‚531380.8984375 disk.md0.out‚ 1077.296875# memory_percent‚38.2279634475708 disk.sdb.out‚ 1117.80859375 disk.sda.out‚ 1117.80859375 disk.nvme0n1.in‚ 329972.65625 disk.sda1.out‚0$ disk./.usageGB‚2974.816349029541" disk.nvme1n1.out‚ 65142.0078125 disk.dm-0.out‚125308.6328125 disk.sda2.out‚ 1117.80859375 disk.sdb2.in‚5.40625 disk.md0.in‚ 10.97265625" proc.memory.rssMB‚ 3031.2421875 network.sent‚ 2.2799119e+07 network.recv‚ 3.47238e+06 disk.nvme0n1.out‚ 60166.625 disk.md0p1.in‚ 10.97265625 disk.rbd0.in‚0.015625&õÒ‘%‚(:† “æçÔ˧é‡ gpu.0.gpu‚37.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚104.494' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚26.1235 gpu.0.smClock‚1215 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤Тu‘%‚(:è “æçÔÀ³ˆ disk.rbd0.in‚0.015625 disk.sdb.out‚ 1119.37109375 disk.sda1.in‚0 disk.md0p0ûþâþ1.out‚ 1078.8046875$ memory_percent‚38.22641372680664 disk.rbd0.out‚ 2.80859375 disk.nvme1n1.out‚ 65142.65625 disk.dm-0.out‚125453.2109375 disk.sdb2.out‚ 1119.37109375 disk.md0.out‚ 1078.8046875 cpu‚25.201835345748115 disk.md0p1.in‚ 10.97265625" proc.memory.rssMB‚ 3031.2421875 disk.rbd2.in‚0 disk.nvme0n1.in‚ 329972.65625 disk.sda1.out‚0 disk.sdb2.in‚5.40625" disk.nvme1n1.in‚201408.2421875 disk.sdb1.out‚0 network.sent‚ 2.2805735e+07 disk.sdb.in‚5.40625) disk./.usagePercent‚83.17782320189116 disk.rbd2.out‚ 2.74609375 disk.sda.in‚ 5.56640625" disk.nvme0n1.out‚ 60310.5546875 disk.sda2.in‚ 5.56640625 disk.sda2.out‚ 1119.37109375 disk.md0.in‚ 10.97265625( proc.memory.availableMB‚ 10120.984375 network.recv‚ 3.47545e+06% disk./.usageGB‚2974.8418502807617 disk.sdb1.in‚0 disk.sda.out‚ 1119.37109375 disk.dm-0.in‚531380.8984375) proc.memory.percent‚18.50123405456543 proc.cpu.threads‚17kF„’%‚(:ù ¢æçÔʉø‡ disk.rbd2.in‚0! disk.nvme1n1.out‚ 65142.828125! disk.sdb.out‚1121.32861328125" proc.memory.rssMB‚ 3031.2421875) proc.memory.percent‚18.50123405456543 cpu‚25.201615226016887 proc.cpu.threads‚17 disk.nvme0n1.in‚ 329972.65625 network.sent‚ 2.2809852e+07$ disk./.usageGB‚2974.907157897949 disk.sdb2.in‚5.40625 disk.md0.in‚ 10.97265625 disk.rbd0.out‚ 2.80859375) disk./.usagePercent‚83.17964923020106# disk.nvme0n1.out‚60384.87109375! disk.dm-0.out‚125527.69921875 network.recv‚ 3.480088e+06 disk.sdb.in‚5.40625 disk.sda1.out‚0 disk.md0.out‚ 1080.70703125 disk.rbd2.out‚ 2.74609375 disk.sdb1.out‚0 disk.md0p1.out‚ 1080.70703125 disk.rbd0.in‚0.015625! disk.sda.out‚1121.32861328125 disk.sda2.in‚ 5.56640625" disk.sda2.out‚1121.32861328125" disk.nvme1n1.in‚201408.2421875 disk.sdb1.in‚0 disk.sda.in‚ 5.56640625 disk.dm-0.in‚531380.8984375 disk.sda1.in‚0 disk.md0p1.in‚ 10.97265625$ memory_percent‚38.22660446166992( proc.memory.availableMB‚ 10120.953125" disk.sdb2.out‚1121.32861328125ãFœ“%‚(:‘ ¢æçÔþ§Þ‡ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›;}ñ4”%/ _wandbruntime‚10206  _runtime‚10206YR“Ê•%Ä _step‚709 ' lr-AdamW/pg1‚0.00018724783848988032 ' lr-AdamW/pg2‚0.00018724783848988032 trainer/global_step‚8649 % _timestamp‚1.7884741529940372e+09 _runtime‚10206.179068907çóYêÏ–%É _runtime‚10206.179068907 _step‚709 ' lr-AdamW/pg1‚0.00018724783848988032 ' lr-AdamW/pg2‚0.00018724783848988032 trainer/global_step‚8649 % _timestamp‚1.7884741529940372e+09Åo)Ot4—%/ _wandbruntime‚10206  _runtime‚10206´2*ï'˜%éO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚353100916126720 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 161170237 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚391463111988224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 235395499 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚370300503910400 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 400270200 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884741529949589e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 235421843 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 400296546 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚353100934129664 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚353100916126720 _runtime‚10206.180370823 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚353073805986104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17199569780736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98597780 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚335923003186512 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17199569780736 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17199336112128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚667 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98597782 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 _step‚710 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚373972997654016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 trainer/global_step‚8649 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 164874703 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 235421843 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚353073559965660 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚370300503910400 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17150802799592 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17489823714816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 164874703 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17490114334208 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚335922986147152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 164874701 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚370300252238848 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62572455 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136œ^vôô'™%îO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884741529949589e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 235421843 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 400296546 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚353100934129664 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚353100916126720 _runtime‚10206.180370823 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚353073805986104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17199569780736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98597780 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚335923003186512 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17199569780736 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17199336112128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚667 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98597782 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 _step‚710 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚373972997654016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 trainer/global_step‚8649 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 164874703 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 235421843 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚353073559965660 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚370300503910400 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17150802799592 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17489823714816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 164874703 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17490114334208 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚335922986147152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 164874701 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚370300252238848 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62572455 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚353100916126720 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 161170237 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚391463111988224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 235395499 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚370300503910400 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 400270200Æç(÷/4š%/ _wandbruntime‚10207  _runtime‚10207ö8²g}#›%òM XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚391506786452480 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 400341539 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚353140202163712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚370341502301696 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62578817 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 161187399 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17152761288786 $ _timestamp‚1.788474154208135e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚353140220166656 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 161188072 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17152532307494 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 235422191 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 400315167 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17201300137984 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 _runtime‚10207.394242721 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚374014681624064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚374014657683968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17201300137984 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 164892978 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚335960516504576 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 235422191 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚370341753986560 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚673 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚391506786452480 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 400341539 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚353140202163712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚370341502301696 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62578817 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 161187399 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17152761288786 $ _timestamp‚1.788474154208135e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚353140220166656 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 161188072 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17152532307494 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 235422191 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 400315167 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17201300137984 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 _runtime‚10207.394242721 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚374014681624064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚374014657683968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17201300137984 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 164892978 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚335960516504576 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 235422191 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚370341753986560 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚673 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚355065235979264 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚355065217976320 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 402548248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚99137880 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚372362769795072 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17297785487360 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 165788453 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62919607 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚10271.241417601 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 236733451 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17589767494144 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚337799521057712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚376057196156416 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17248740840100 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚372362769795072 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 162056800 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62918920 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚8699 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 165788451 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚355065235979264 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚355048261897812 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17248511859016 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚355065217976320 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17297785487360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚393647266852864 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 327142400 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚393646939710464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚337799504018352 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _step‚714 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17297551818752 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚372363021466624 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344”hüù­%íO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚355065217976320 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 402548248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚99137880 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚372362769795072 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17297785487360 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 165788453 H?DeviceStatsMoÇgM1únitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62919607 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚10271.241417601 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 236733451 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17589767494144 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚337799521057712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚376057196156416 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17248740840100 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚372362769795072 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 162056800 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62918920 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚8699 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 165788451 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚355065235979264 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚355048261897812 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17248511859016 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚355065217976320 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17297785487360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚393647266852864 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 327142400 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚393646939710464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚337799504018352 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _step‚714 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17297551818752 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚372363021466624 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17297551818752 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚355048015877368 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 236759795 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚355065235979264 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0Ê 3#X4®%/ _wandbruntime‚10272  _runtime‚10272R‚„Êø&¯%òM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚372404271542784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 165806728 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 236786513 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚99148684 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17250470348002 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 162074472 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚337837034375776 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚8699 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚355104522016256 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17299515844608 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚393690932000256 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17592051873792 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 236760143 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 162075146 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.788474219257399e+09 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚393690627939840 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚376098856186368 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚372404271542784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚10272.443984332 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 402593241 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 402593241 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62926462 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚376098880126464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17299515844608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62925790 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280120320 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚715 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚99148682 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 236760143 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17591771753472 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17299749526528 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 304060416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 165806726 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚372404019857920 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 236786513 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚674 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚355104522016256 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚372404271542784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚10272.443984332 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 402593241 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 402593241 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62926462 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚376098880126464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17299515844608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62925790 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280120320 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚715 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚99148682 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 236760143 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17591771753472 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17299749526528 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 304060416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 165806726 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚372404019857920 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 236786513 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚674 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚355104522016256 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 165806728 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 236786513 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚99148684 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17250470348002 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 162074472 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚337837034375776 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚8699 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚355104522016256 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17299515844608 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚393690932000256 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17592051873792 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 236760143 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 162075146 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0Ëa¾±4±%/ _wandbruntime‚10272  _runtime‚10272¬$í®`²%Ú 0loss/FBt4/assignment_loss‚0.3042008560730422 /loss/FRt1/detection_loss‚2.6780563768218544 1loss/SRqqt3/detection_loss‚0.8606923990619352 %loss/total_loss‚40.47382266094024 $ _timestamp‚1.788474219261935e+09 2loss/SRqqt4/assignment_loss‚1.2000103427562863 1loss/FBt1/assignment_loss‚0.03870253544438735 1loss/FBt3/assignment_loss‚0.08713030734040716 /loss/FRt3/detection_loss‚1.3161461987299845 0loss/SRqqt4/detection_loss‚2.597336928049723 .loss/FBt2/detection_loss‚2.081640273332596 /loss/FBt3/detection_loss‚0.5113506456750825 trainer/global_step‚8699 _runtime‚10272.446883205 0loss/FRt1/assignment_loss‚2.1814326836782345 0loss/FRt3/assignment_loss‚1.0968319319188595 2loss/SRqqt2/assignment_loss‚1.3253010308434223 1loss/SRqqt2/detection_loss‚2.4126086132279756 _step‚716 1loss/SRqqt1/detection_loss‚1.7443754843303136 2loss/SRqqt3/assignment_loss‚1.1128820650903333 2loss/FBt2/assignment_loss‚0.021433725972327976 /loss/FRt2/detection_loss‚2.4105015997468544 epoch‚7 0loss/FRt2/assignment_loss‚3.9585909027802315 .loss/FBt1/detection_loss‚2.548856576283773 2loss/SRqqt1/assignment_loss‚1.2569749829847188 /loss/FRt4/detection_loss‚3.3445624056316556 /loss/FBt4/detection_loss‚2.2804484516382217 0loss/FRt4/assignment_loss‚3.1037553435280207MûOe³%ß 2loss/SRqqt3/assignment_loss‚1.1128820650903333 2loss/FBt2/assignment_loss‚0.021433725972327976 /loss/FRt2/detection_loss‚2.4105015997468544 epoch‚7 0loss/FRt2/assignment_loss‚3.9585909027802315 .loss/FBt1/detection_loss‚2.548856576283773 2loss/SRqqt1/assignment_loss‚1.2569749829847188 /loss/FRt4/detection_loss‚3.3445624056316556 /loss/FBt4/detection_loss‚2.2804484516382217 0loss/FRt4/assignment_loss‚3.1037553435280207 0loss/FBt4/assignment_loss‚0.3042008560730422 /loss/FRt1/detection_loss‚2.6780563768218544 1loss/SRqqt3/detection_loss‚0.8606923990619352 %loss/total_loss‚40.47382266094024 $ _timestamp‚1.788474219261935e+09 2loss/SRqqt4/assignment_loss‚1.2000103427562863 1loss/FBt1/assignment_loss‚0.03870253544438735 1loss/FBt3/assignment_loss‚0.08713030734040716 /loss/FRt3/detection_loss‚1.3161461987299845 0loss/SRqqt4/detection_loss‚2.597336928049723 .loss/FBt2/detection_loss‚2.081640273332596 /loss/FBt3/detection_loss‚0.5113506456750825 trainer/global_step‚8699 _runtime‚10272.446883205 0loss/FRt1/assignment_loss‚2.1814326836782345 0loss/FRt3/assignment_loss‚1.0968319319188595 2loss/SRqqt2/assignment_loss‚1.3253010308434223 1loss/SRqqt2/detection_loss‚2.4126086132279756 _step‚716 1loss/SRqqt1/detection_loss‚1.7443754843303136ÌZ’–ø´%‚(:… íæçԤ쾆 gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚32.0 gpu.0.powerWatts‚86.502' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚21.6255 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c¤³è{µ%‚(:ð íæçÔ¯×Ö‡" disk.sda2.out‚1129.70556640625 disk.md0p1.out‚ 1088.73828125 disk.sdb1.in‚0$ disk./.usageGB‚2974.817440032959 disk.dm-0.out‚ 126423.078125 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.nvme0n1.in‚ 329972.65625 disk.md0p1.in‚ 10.97265625 disk.nvme1n1.out‚ 65143.53125 disk.sdb.in‚5.40625! disk.sda.out‚1129.70556640625 disk.sda1.in‚0) proc.memory.percent‚18.50123405456543 network.recv‚ 3.503982e+06 disk.md0.in‚ 10.97265625 proc.cpu.threads‚17 disk.sda2.in‚ 5.56640625 disk.sdb2.in‚5.40625" disk.sdb2.out‚1129.70556640625 disk.sdb1.out‚0( disk./.usagePercent‚83.1771406811461 disk.rbd0.out‚ 2.8203125! disk.nvme0n1.out‚ 61279.546875 disk.dm-0.in‚531380.8984375% memory_percent‚39.273929595947266 cpu‚25.200416395387563! disk.sdb.out‚1129.70556640625 network.sent‚ 2.294115e+07 disk.rbd2.out‚ 2.7578125" disk.nvme1n1.in‚201408.2421875 disk.sda.in‚ 5.56640625" proc.memory.rssMB‚ 3031.2421875 disk.sda1.out‚0 disk.md0.out‚ 1088.73828125' proc.memory.availableMB‚ 9949.359375…àÔ•“¶%‚(:† üæçÔɇ gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686!Mþ914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚136.466' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.1165 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽœÀ±ƒ·%‚(:ø üæçÔÅúø‡! disk.sdb.out‚1131.71728515625! disk.dm-0.out‚126496.12890625 disk.sda1.out‚0 disk.rbd2.in‚0" disk.sdb2.out‚1131.71728515625 disk.rbd0.in‚0.015625 disk.sda1.in‚0 disk.md0p1.out‚ 1090.68359375 disk.sdb1.in‚0! disk.sda.out‚1131.71728515625 proc.cpu.threads‚17 disk.sda2.in‚ 5.56640625" proc.memory.rssMB‚ 3031.2421875) proc.memory.percent‚18.50123405456543) disk./.usagePercent‚83.14818938627018" disk.nvme1n1.in‚201408.2421875 disk.sdb1.out‚0 disk.sda.in‚ 5.56640625 cpu‚25.20019643024401# disk.nvme1n1.out‚65143.85546875 network.recv‚ 3.51077e+06 disk.sdb2.in‚5.40625 disk.nvme0n1.in‚ 329972.65625" disk.nvme0n1.out‚ 61352.2734375 disk.md0p1.in‚ 10.97265625 network.sent‚ 2.3051101e+07 disk.rbd2.out‚ 2.7578125 disk.sdb.in‚5.40625 disk.dm-0.in‚531380.8984375" disk.sda2.out‚1131.71728515625% memory_percent‚39.273881912231445( proc.memory.availableMB‚ 9949.3671875% disk./.usageGB‚2973.7820014953613 disk.md0.in‚ 10.97265625 disk.md0.out‚ 1090.68359375 disk.rbd0.out‚ 2.8828125ÝíèU¸%‚(:ö ‹ççÔ’Ö؇ disk.rbd2.in‚0 disk.sda1.out‚0 disk.sda2.in‚ 5.56640625 disk.rbd0.in‚0.015625& proc.memory.availableMB‚ 9949.53125) proc.memory.percent‚18.50123405456543) disk./.usagePercent‚83.14910624020429" disk.nvme1n1.in‚201411.9296875 disk.sdb1.out‚0 disk.md0p1.out‚ 1091.96484375 disk.md0.out‚ 1091.96484375 cpu‚25.199931479834976 disk.dm-0.in‚531386.42578125! disk.sdb2.out‚1133.0712890625$ memory_percent‚39.27288055419922# disk.nvme1n1.out‚65144.00390625 disk.sdb.out‚1133.0712890625 disk.sdb1.in‚0! disk.nvme0n1.out‚ 61545.515625% disk./.usageGB‚2973.8147926330566 disk.sdb.in‚5.40625 disk.md0.in‚ 10.97265625 disk.rbd2.out‚ 2.8203125 disk.sda.out‚1133.0712890625# disk.nvme0n1.in‚329974.49609375 disk.sda1.in‚0 disk.rbd0.out‚ 2.8828125 disk.sda.in‚ 5.56640625! disk.dm-0.out‚126689.51953125! disk.sda2.out‚1133.0712890625 disk.md0p1.in‚ 10.97265625 disk.sdb2.in‚5.40625" proc.memory.rssMB‚ 3031.2421875 network.sent‚ 2.3054197e+07 proc.cpu.threads‚17 network.recv‚ 3.514737e+06 †8E’¹%‚(:‡ ‹ççÔÛÅ̇ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚105.111' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚26.27775 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’º%‚(:‡ šççÔߣž‡ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚137.693' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.42325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CÞm‡»%‚(:ü šççÔŽ”Ô‡ disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 9949.28515625 disk.sdb.in‚5.40625! disk.sdb.out‚1134.62451171875! disk.sda.out‚1134.62451171875! disk.dm-0.out‚126769.91796875) proc.memory.percent‚18.50123405456543) disk./.usagePercent‚83.15520412948622 disk.md0.out‚ 1093.4609375 disk.rbd2.out‚ 2.8203125" disk.nvme1n1.in‚201411.9296875 disk.nvme0n1.out‚ 61625.53125 disk.dm-0.in‚531386.42578125" disk.sda2.out‚1134.62451171875" disk.sdb2.out‚1134.62451171875 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.sda.in‚ 5.56640625 disk.md0p1.out‚ 1093.4609375 cpu‚25.19966611556023 network.recv‚ 3.519192e+06 disk.sdb1.out‚0% disk./.usageGB‚2974.0328826904297 disk.md0.in‚ 10.97265625 disk.sdb1.in‚0 disk.md0p1.in‚ 10.97265625" proc.memory.rssMB‚ 3031.2421875 proc.cpu.threads‚17 network.sent‚ 2.3060664e+07 disk.rbd0.out‚ 2.9619140625# disk.nvme1n1.out‚65144.38671875 disk.sda2.in‚ 5.56640625 disk.sdb2.in‚5.40625$ memory_percent‚39.27438259124756# disk.nvme0n1.in‚329974.49609375Y#ÝS4¼%/ _wandbruntime‚10327  _runtime‚10327a(áoǽ%Á & lr-AdamW/pg1‚0.0001773699162474874 & lr-AdamW/pg2‚0.0001773699162474874 trainer/global_step‚8749 $ _timestamp‚1.788474274656388e+09 _runtime‚10327.841456492 _step‚717"ΰ:̾%Æ $ _timestamp‚1.788474274656388e+09 _runtime‚10327.841456492 _step‚717 & lr-AdamW/pg1‚0.0001773699162474874 & lr-AdamW/pg2‚0.0001773699162474874 trainer/global_step‚8749Í­1ïÖ4¿%/ _wandbruntime‚10327  _runtime‚10327ÀÞYï'À%éO \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17396001193984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 238071403 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 238097747 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 162948426 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374425287351296 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚357029537828864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 404799950 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚718 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63270444 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 162947708 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚716 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚357022717809520 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚339676038928912 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 166702203 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚99677982 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚374425539022848 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚357022471789076 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17346678880608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 % _timestamp‚1.7884742746575975e+09 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17689926815232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚339676021889552 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚374425287351296 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚378141394658816 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17689642487296 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 166702201 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17395767525376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63269728 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚357029519825920 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚395831013206016 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17346449899524 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚374425539022848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚378141370718720 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚357029519825920 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309j"$eô'Á%îO C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚374425287351296 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚378141394658816 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17689642487296 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 166702201 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17395767525376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63269728 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚357029519825920 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚395831013206016 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17346449899524 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚374425539022848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚378141370718720 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚357029519825920 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17396001193984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 238071403 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 238097747 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 162948426 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374425287351296 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚357029537828864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 404799950 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚718 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63270444 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 162947708 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚716 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚357022717809520 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚339676038928912 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 166702203 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚99677982 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚374425539022848 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚357022471789076 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17346678880608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 % _timestamp‚1.7884742746575975e+09 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17689926815232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚339676021889552ÎsìÐ4Â%/ _wandbruntime‚10328  _runtime‚10328èC,ßÃ%óM PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚374466789099008 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 166720476 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17397731551232 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _step‚719 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚357061960635486 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 238098095 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚357068805862912 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17397965233152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚99688782 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17691636341248 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17397965233152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 % _timestamp‚1.7884742756309736e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚374466789099008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 166720478 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 238124465 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 404818571 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 238124465 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 166720478 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63276389 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚395874691030016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 162965171 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17397731551232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚378183054688768 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚374466537414144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 404844943 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚339713552246976 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚395875005576192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚378183078628864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 404818571 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚99688784 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚339713569286336 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608½»öþ&Ä%øM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17397731551232 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _step‚719 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚357061960635486 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 238098095 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚357068805862912 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17397965233152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚99688782 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17691636341248 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17397965233152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 % _timestamp‚1.7884742756309736e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚374466789099008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 166720478 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 238124465 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 404818571 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 238124465 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 166720478 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63276389 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚395874691030016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 162965171 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17397731551232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚378183054688768 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚374466537414144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 404844943 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚339713552246976 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚395875005576192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚378183078628864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 404818571 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚99688784 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚339713569286336 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚374466789099008 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 166720476ÏIÄÇË4Å%/ _wandbruntime‚10328  _runtime‚10328j°~)\Æ%Ö 2loss/SRqqt1/assignment_loss‚1.6660922629137833 2loss/SRqqt2/assignment_loss‚1.3761147186160088 1loss/SRqqt4/assignment_loss‚1.676858055735788 1loss/FBt1/assignment_loss‚0.09241935439461356 1loss/FBt3/assignment_loss‚0.10561842680355515 /loss/FRt4/detection_loss‚3.0735364120740156 0loss/SRqqt2/detection_loss‚2.089848135198866 .loss/FBt1/detection_loss‚3.001823306083679 0loss/FRt1/assignment_loss‚3.1976784113291146 /loss/FRt3/detection_loss‚1.3602678727890765 1loss/SRqqt3/detection_loss‚0.9310228656977415 epoch‚7 0loss/FRt2/assignment_loss‚3.1558278636498884 1loss/SRqqt3/assignment_loss‚0.971147172005765 /loss/FBt4/detection_loss‚2.8866804133762014 $loss/total_loss‚42.9307803996485 trainer/global_step‚8749 /loss/FRt4/assignment_loss‚3.955873931829746 2loss/FBt2/assignment_loss‚0.062322462143033164 2loss/FBt4/assignment_loss‚0.037700809464265135 .loss/FRt1/detection_loss‚2.573707667556969 /loss/FBt3/detection_loss‚0.4834668198802873 _step‚720 /loss/FRt2/detection_loss‚2.2558128365847305 1loss/SRqqt4/detection_loss‚2.4886595818304245 _runtime‚10328.818237536 0loss/FRt3/assignment_loss‚1.1932801817144667 /loss/SRqqt1/detection_loss‚2.15229614244567 .loss/FBt2/detection_loss‚2.142724695530805 % _timestamp‚1.7884742756330807e+09á0ÒaÇ%Û 0loss/FRt1/assignment_loss‚3.1976784113291146 /loss/FRt3/detection_loss‚1.3602678727890765 1loss/SRqqt3/detection_loss‚0.9310228656977415 epoch‚7 0loss/FRt2/assignment_loss‚3.1558278636498884 1loss/SRqqt3/assignment_loss‚0.971147172005765 /loss/FBt4/detection_loss‚2.8866804133762014 $loss/total_loss‚42.9307803996485 trainer/global_step‚8749 /loss/FRt4/assignment_loss‚3.955873931829746 2loss/FBt2/assignment_loss‚0.062322462143033164 2loss/FBt4/assignment_loss‚0.037700809464265135 .loss/FRt1/detection_loss‚2.573707667556969 /loss/FBt3/detection_loss‚0.4834668198802873 _step‚720 /loss/FRt2/detection_loss‚2.2558128365847305 1loss/SRqqt4/detection_loss‚2.4886595818304245 _runtime‚10328.818237536 0loss/FRt3/assignment_loss‚1.1932801817144667 /loss/SRqqt1/detection_loss‚2.15229614244567 .loss/FBt2/detection_loss‚2.142724695530805 % _timestamp‚1.7884742756330807e+09 2loss/SRqqt1/assignment_loss‚1.6660922629137833 2loss/SRqqt2/assignment_loss‚1.3761147186160088 1loss/SRqqt4/assignment_loss‚1.676858055735788 1loss/FBt1/assignment_loss‚0.09241935439461356 1loss/FBt3/assignment_loss‚0.10561842680355515 /loss/FRt4/detection_loss‚3.0735364120740156 0loss/SRqqt2/detection_loss‚2.089848135198866 .loss/FBt1/detection_loss‚3.001823306083679ÐLôµ<‘È%‚(:† ©ççÔ’¸«ˆ gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚138.642' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.6605 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“⤋É%‚(:€  ©ççÔëµÃˆ disk.sdb.in‚5.40625! disk.sdb.out‚1136.48974609375" proc.memory.rssMB‚ 3031.2421875 network.sent‚ 2.306653e+07 disk.rbd2.in‚0" disk.nvme1n1.in‚201411.9296875! disk.sda.out‚1136.48974609375! disk.dm-0.out‚126925.06640625 disk.md0p1.in‚ 10.97265625' proc.memory.availableMB‚ 9949.515625$ disk./.usageGB‚2974.284412384033" disk.sda2.out‚1136.48974609375 disk.md0.out‚ 1095.26953125 cpu‚25.199372822624753 disk.rbd2.out‚ 2.8994140625# disk.nvme1n1.out‚65144.49609375 disk.sda1.in‚0 proc.cpu.threads‚17 network.recv‚ 3.523757e+06 disk.sdb1.in‚0 disk.sda1.out‚0 disk.sdb2.in‚5.40625$ memory_percent‚39.27297592163086 disk.sdb1.out‚0# disk.nvme0n1.in‚329974.49609375 disk.dm-0.in‚531386.42578125 disk.sda2.in‚ 5.56640625 disk.md0p1.out‚ 1095.26953125) proc.memory.percent‚18.50123405456543) disk./.usagePercent‚83.16223700499273 disk.sda.in‚ 5.56640625" disk.nvme0n1.out‚ 61780.5703125 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 2.9619140625" disk.sdb2.out‚1136.48974609375 disk.md0.in‚ 10.97265625뮼’Ê%‚(:‡ ¸ççÔ¸ª³‡ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚100.759' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.18975 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›’èxË%‚(:í ¸ççÔæ∠disk.sdb2.out‚ 1138.2578125 disk.rbd2.in‚0# disk.nvme1n1.out‚65144.65234375 disk.md0p1.in‚ 10.97265625) proc.memory.percent‚18.50123405456543 disk.md0.in‚ 10.97265625 disk.sda.in‚ 5.56640625 disk.rbd0.out‚ 3.02490234375" disk.nvme1n1.in‚201411.9296875 disk.sdb1.in‚0 disk.sda.out‚ 1138.2578125! disk.nvme0n1.out‚ 61935.734375" proc.memory.rssMB‚ 3031.2421875 disk.rbd0.in‚0.015625 disk.dm-0.in‚531386.42578125! disk.dm-0.out‚127080.38671875 disk.md0p1.out‚ 1096.96875 disk.rbd2.out‚ 2.8994140625$ memory_percent‚39.27445411682129 network.sent‚ 2.3174285e+07 network.recv‚ 3.530617e+06 disk.md0.out‚ 1096.96875 disk.sdb.in‚5.40625 cpu‚25.199086135141055 disk.sdb2.in‚5.40625# disk.nvme0n1.in‚329974.49609375 disk.sda1.out‚0 disk.sda2.in‚ 5.56640625 disk.sda2.out‚ 1138.2578125( proc.memory.availableMB‚ 9949.2734375 proc.cpu.threads‚17) disk./.usagePercent‚83.16220575345683% disk./.usageGB‚2974.2832946777344 disk.sdb.out‚ 1138.2578125 disk.sdb1.out‚0 disk.sda1.in‚0Ÿ|WÕ’Ì%‚(:‡ ÇççÔ¤«–‡ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚143.595' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.89875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[;øPÍ%‚(:‚  ÇççÔ®½ä‡" disk.nvme1n1.out‚ 65144.7421875! disk.sdb.out‚1139.87255859375 network.sent‚ 2.3179618e+07 disk.sda1.in‚0! disk.sda.out‚1139.87255859375 disk.sdb.in‚5.40625 disk.md0p1.out‚ 1098.51953125# disk.nvme0n1.in‚329974.49609375 disk.dm-0.in‚531386.42578125 disk.dm-0.out‚127193.9453125) proc.memory.availableMB‚ 9949.51953125$ disk./.usageGB‚2974.157958984375 disk.sdb2.in‚5.40625 disk.rbd0.out‚ 3.02490234375 disk.sdb1.in‚0$ memory_percent‚39.27295207977295" proc.memory.rssMB‚ 3031.2421875 disk.md0p1.in‚ 10.97265625 disk.rbd0.in‚0.015625" disk.sda2.out‚1139.87255859375" disk.nvme1n1.in‚201411.9296875 disk.sdb1.out‚0) proc.memory.percent‚18.50123405456543" disk.sdb2.out‚1139.87255859375 disk.md0.in‚ 10.97265625 disk.rbd2.out‚ 2.96240234375 network.recv‚ 3.534244e+06) disk./.usagePercent‚83.15870131501345! disk.nvme0n1.out‚ 62049.203125 disk.sda2.in‚ 5.56640625 disk.sda.in‚ 5.56640625 disk.sda1.out‚0 disk.md0.out‚ 1098.51953125 cpu‚25.198825418633575 proc.cpu.threads‚17 disk.rbd2.in‚0ݳ<ä’Î%‚(:‡ ÖççÔ§‹¦† gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚140.145' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.03625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 뽜Ï%‚(:ô ÖççÔùÁô†" disk.nvme1n1.in‚201411.9296875 disk.sdb2.in‚5.40625! disk.dm-0.out‚127254.92578125 disk.sda2.in‚ 5.56640625" proc.memory.rssMB‚ 3031.2421875 network.recv‚ 3.53867e+06# disk.nvme0n1.out‚62110.01171875$ memory_percent‚39.27311897277832( proc.memory.availableMB‚ 9949.4921875 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 3.087890625 disk.rbd2.out‚ 2.96240234375 disk.sda.in‚ 5.56640625 disk.dm-0.in‚531386.42578125 disk.md0p1.in‚ 10.97265625" disk.nvme1n1.out‚ 65144.9140625 disk.sdb1.out‚0# disk.nvme0n1.in‚329974.49609375 disk.sda1.in‚0 proc.cpu.threads‚17 network.sent‚ 2.3186214e+07) disk./.usagePercent‚83.14394963011834 disk.rbd2.in‚0 disk.sdb.out‚1141.263671875 cpu‚25.198543457058328 disk.md0.out‚ 1099.84375 disk.sdb.in‚5.40625 disk.sda1.out‚0 disk.sda2.out‚1141.263671875% disk./.usageGB‚2973.6303672790527 disk.sdb1.in‚0 disk.md0p1.out‚ 1099.84375) proc.memory.percent‚18.50123405456543 disk.sdb2.out‚1141.263671875 disk.md0.in‚ 10.97265625 disk.sda.out‚1141.263671875ø¡Ö.4Ð%/ _wandbruntime‚10383  _runtime‚10383çÁL,ÈÑ% & lr-AdamW/pg2‚0.0001725038585803381 trainer/global_step‚8799 % _timestamp‚1.7884743298184469e+09 _runtime‚10383.003422911 _step‚721 & lr-AdamW/pg1‚0.0001725038585803381L{ÞSÍÒ%Ç & lr-AdamW/pg1‚0.0001725038585803381 & lr-AdamW/pg2‚0.0001725038585803381 trainer/global_step‚8799 % _timestamp‚1.7884743298184469e+09 _runtime‚10383.003422911 _step‚721Ñ ³ä«4Ó%/ _wandbruntime‚10383  _runtime‚10383lóŒ.ð'Ô%êO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚398015470819328 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318753792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚398015152065536 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚376487804907520 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17494216900608 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 167615951 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚380225569221120 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 167615951 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚358993821675520 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17444387940032 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚376488056579072 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 167615953 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 239435699 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17493983232000 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294813696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚689 trainer/global_step‚8799 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17494216900608 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17789582844416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚358997173721228 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63617174 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚376487804907520 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 100218082 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚380225593161216 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 239409355 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚358993839678464 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 407051652 _step‚722 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63617863 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚358996927700784 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚341552539760752 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17789877658112 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚358993839678464 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 239409355 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚376488056579072 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 163835945 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 100218080 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 163835254 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17493983232000 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 _runtime‚10383.004338835 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚376487804907520 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17494216900608 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 167615951 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚380225569221120 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 167615951 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚358993821675520 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17444387940032ÒQ‚Ãÿ4Ö%/ _wandbruntime‚10383  _runtime‚10383Ð.Ê=û&×%õM ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17496180939776 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100228884 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚10383.956359345 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 167634228 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 239462417 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚359033125715456 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 163853533 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17791869389312 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚341590087157536 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚380267277131264 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚359033125715456 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 167634226 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 239436047 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63624651 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 % _timestamp‚1.7884743307708266e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63625333 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚398059146520576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚359036662567846 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17495947257856 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚376529306655232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 trainer/global_step‚8799 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17495947257856 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17446346429018 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 407096645 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17791576686080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚359033107712512 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚341590070118176 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _step‚723 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 407096645 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 167634226 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 239436047 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚380267253191168 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17496180939776 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚359036416547194 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 407070273 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚376529306655232 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920½`X'Ø%úM BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 407096645 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 167634226 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 239436047 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚380267253191168 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17496180939776 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚359036416547194 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 407070273 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚376529306655232 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17496180939776 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100228884 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚10383.956359345 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 167634228 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 239462417 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚359033125715456 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 163853533 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17791869389312 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚341590087157536 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚380267277131264 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚359033125715456 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 167634226 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 239436047 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63624651 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 % _timestamp‚1.7884743307708266e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63625333 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚398059146520576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚359036662567846 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17495947257856 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚376529306655232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 trainer/global_step‚8799 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17495947257856 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17446346429018 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 407096645 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17791576686080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚359033107712512 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚341590070118176 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _step‚723 FÔFkƒÝ%‚(:ø åççÔÖËïˆ disk.rbd2.in‚0 disk.sdb1.in‚0# disk.nvme0n1.out‚62295.92578125) proc.memory.percent‚18.50123405456543 disk.sda.out‚1143.0908203125" proc.memory.rssMB‚ 3031.2421875 disk.sda2.in‚ 5.56640625 disk.md0p1.in‚ 10.97265625 disk.md0.out‚ 1101.58984375 disk.rbd0.in‚0.015625 disk.sdb.in‚5.40625 disk.sdb1.out‚0# disk.nvme0n1.in‚329974.49609375 disk.sdb2.in‚5.40625! disk.sdb2.out‚1143.0908203125( disk./.usagePercent‚83.1486808780022 disk.dm-0.in‚531386.42578125! disk.dm-0.out‚127441.05859375 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0p1.out‚ 1101.58984375% disk./.usageGB‚2973.7995796203613! disk.sda2.out‚1143.0908203125 disk.md0.in‚ 10.97265625 disk.rbd0.out‚ 3.087890625" disk.nvme1n1.out‚ 65145.1328125 disk.sda.in‚ 5.56640625$ memory_percent‚39.27345275878906 proc.cpu.threads‚17 network.sent‚ 2.3191946e+07 network.recv‚ 3.542684e+06 disk.rbd2.out‚ 3.025390625 disk.sdb.out‚1143.0908203125% proc.memory.availableMB‚ 9949.4375" disk.nvme1n1.in‚201411.9296875 cpu‚25.19829589468401´±S;‘Þ%‚(:† ôççÔ¨Žž† gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚144.821' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.20525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00«ðjˆß%‚(:ý ôççÔ˜†‡ disk.nvme0n1.out‚ 62296.21875 disk.md0.in‚ 10.97265625 proc.cpu.threads‚17 network.recv‚ 3.549987e+06! disk.sda.out‚1144.67333984375 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 3.087890625) proc.memory.percent‚18.50123405456543 network.sent‚ 2.3299955e+07 disk.md0p1.in‚ 10.97265625 disk.sdb.in‚5.40625! disk.sdb.out‚1144.67333984375 disk.sda.in‚ 5.56640625# disk.nvme0n1.in‚329974.49609375) disk./.usagePercent‚83.14963932954385 disk.sda1.in‚0" disk.sda2.out‚1144.67333984375 disk.md0p1.out‚ 1103.12890625 disk.dm-0.in‚531386.42578125 disk.dm-0.out‚127510.2734375 disk.sdb2.in‚5.40625" disk.sdb2.out‚1144.67333984375" proc.memory.rssMB‚ 3031.2421875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 3.025390625" disk.nvme1n1.in‚201411.9296875$ memory_percent‚39.27512168884277( proc.memory.availableMB‚ 9949.1640625% disk./.usageGB‚2973.8338584899902 disk.sda2.in‚ 5.56640625 disk.md0.out‚ 1103.12890625" disk.nvme1n1.out‚ 65214.0546875 cpu‚25.198044226812513ÇÇ—Ý‘à%‚(:† ƒèçÔƒï· gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚113.798' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.4495 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡ö·ÿ‡á%‚(:ü ƒèçÔÒÕˆ proc.cpu.threads‚17 disk.sdb2.in‚5.40625 disk.md0.out‚ 1104.671875 disk.rbd2.in‚0 disk.rbd2.out‚ 3.025390625# disk.nvme0n1.out‚62296.27734375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚ 5.56640625% disk./.usageGB‚2973.9590072631836! disk.sda.out‚1146.27392578125 disk.md0p1.in‚ 10.97265625 disk.md0p1.out‚ 1104.671875 disk.dm-0.out‚ 127556.359375" disk.sda2.out‚1146.27392578125 network.recv‚ 3.553525e+06# disk.nvme0n1.in‚329974.49609375 disk.dm-0.in‚531386.42578125 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 3.087890625" disk.nvme1n1.in‚201411.9296875 disk.sdb1.in‚0$ memory_percent‚39.27359580993652( proc.memory.availableMB‚ 9949.4140625) disk./.usagePercent‚83.15313854162116 disk.md0.in‚ 10.97265625 disk.sdb.in‚5.40625) proc.memory.percent‚18.50123405456543" disk.sdb2.out‚1146.27392578125# disk.nvme1n1.out‚65260.08203125! disk.sdb.out‚1146.27392578125 disk.sdb1.out‚0" proc.memory.rssMB‚ 3031.2421875 cpu‚25.197802600606558 network.sent‚ 2.3305231e+07 disk.sda.in‚ 5.56640625É|¸4â%/ _wandbruntime‚10434  _runtime‚10434Žn¹ËÊã%Ä ' lr-AdamW/pg1‚0.00016768775872131944 ' lr-AdamW/pg2‚0.00016768775872131944 trainer/global_step‚8849 % _timestamp‚1.7884743811423726e+09 _runtime‚10434.327545446 _step‚725Ë…ÔÏä%É ' lr-AdamW/pg1‚0.00016768775872131944 ' lr-AdamW/pg2‚0.00016768775872131944 trainer/global_step‚8849 % _timestamp‚1.7884743811423726e+09 _runtime‚10434.327545446 _step‚725Õó]W£4å%/ _wandbruntime‚10434  _runtime‚10434áãYYð'æ%êO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚360958123525120 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚382309767723520 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17542554961624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚8849 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚343429074671312 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63964342 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚400199535960576 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 409277008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚382309791663616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 240747307 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 100758180 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17542325980540 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17592432607232 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304073728 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _runtime‚10434.328958833 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚378550322463744 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚360958141528064 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17592198938624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚360971383612492 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17889744296960 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17889464163328 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚726 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17592432607232 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚360971629632936 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 168529701 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚378550574135296 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚400199231886848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.788474381143732e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚378550322463744 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 240773651 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 409303354 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 100758182 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17592198938624 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚642 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 168529701 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 168529703®É.õ'ç%ïO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚378550574135296 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚400199231886848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.788474381143732e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚378550322463744 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 240773651 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 409303354 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 100758182 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17592198938624 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚642 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 168529701 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 168529703 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚360958123525120 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚382309767723520 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17542554961624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚8849 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚343429074671312 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63964342 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚400199535960576 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 409277008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚382309791663616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 240747307 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 100758180 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17542325980540 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17592432607232 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304073728 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _runtime‚10434.328958833 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚378550322463744 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚360958141528064 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17592198938624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚360971383612492 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17889744296960 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17889464163328 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚726 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17592432607232 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚360971629632936 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 168529701Ö{\ïø4è%/ _wandbruntime‚10435  _runtime‚10435Èq7¾é%õM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17594396646400 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 240773999 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 409348347 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚378591572526592 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17594162964480 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884743821201189e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 409321975 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17891457754624 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚343466605028736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚10435.305726886 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚360997427565056 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚360997427565056 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 409348347 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17544284469526 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 168547976 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17891748360704 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 240773999 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 164739915 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚360997409562112 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚360997409562112 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚361010872458902 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 168547976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63970933 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 _step‚727 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17594396646400 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 409321975 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚343466587989376 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 168547978 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 240800369 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 164740591 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚400242909448192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 34çwKß 7418624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚382351451693568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17594162964480 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚378591824211456 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚382351475633664 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚361011118479554 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100768984 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚378591824211456 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17544513450818 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 240800369 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0Œ$%ï'ê%úM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚378591824211456 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17544513450818 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 240800369 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17594396646400 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 240773999 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 409348347 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚378591572526592 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17594162964480 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884743821201189e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 409321975 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17891457754624 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚343466605028736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚10435.305726886 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚360997427565056 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚360997427565056 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 409348347 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17544284469526 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 168547976 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17891748360704 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 240773999 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 164739915 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚360997409562112 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚360997409562112 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚361010872458902 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 168547976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63970933 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 _step‚727 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17594396646400 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 409321975 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚343466587989376 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 168547978 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 240800369 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 164740591 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚400242909448192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚382351451693568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17594162964480 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚378591824211456 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚382351475633664 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚361011118479554 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100768984 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136×NÝ}4ë%/ _wandbruntime‚10435  _runtime‚10435Knº„[ì%Õ _step‚728 .loss/FBt4/detection_loss‚3.593235452969869 epoch‚7 /loss/SRqqt4/detection_loss‚2.46129727936708 2loss/SRqqt1/assignment_loss‚1.3389962702575657 1loss/SRqqt1/detection_loss‚2.1719399525059595 _runtime‚10435.306841002 /loss/FRt2/assignment_loss‚3.664856642484665 1loss/FBt2/assignment_loss‚0.04343469315164761 0loss/FBt3/assignment_loss‚0.0206913284434285 2loss/FBt4/assignment_loss‚0.023818444771071274 .loss/FBt1/detection_loss‚1.992086071234483 trainer/global_step‚8849 % _timestamp‚1.7884743821221244e+09 0loss/FRt3/assignment_loss‚1.0255450477471222 /loss/FRt4/assignment_loss‚2.580335241297017 2loss/SRqqt3/assignment_loss‚0.9157527719596119 /loss/FRt1/detection_loss‚2.6664972051101574 /loss/FRt4/detection_loss‚2.9005151261454043 1loss/SRqqt2/detection_loss‚2.4066099799596348 .loss/FBt2/detection_loss‚1.966431213542819 1loss/SRqqt4/assignment_loss‚1.632305624966438 0loss/FBt1/assignment_loss‚0.1302797708552882 /loss/FRt2/detection_loss‚1.9435023805466092 /loss/FRt3/detection_loss‚1.4535071426574644 /loss/FBt3/detection_loss‚0.4775688377179442 /loss/FRt1/assignment_loss‚3.243558448903701 2loss/SRqqt2/assignment_loss‚1.4454066180265868 1loss/SRqqt3/detection_loss‚0.8876751905890427 &loss/total_loss‚40.985846735210615&²1¯`í%Ú .loss/FBt1/detection_loss‚1.992086071234483 trainer/global_step‚8849 % _timestamp‚1.7884743821221244e+09 0loss/FRt3/assignment_loss‚1.0255450477471222 /loss/FRt4/assignment_loss‚2.580335241297017 2loss/SRqqt3/assignment_loss‚0.9157527719596119 /loss/FRt1/detection_loss‚2.6664972051101574 /loss/FRt4/detection_loss‚2.9005151261454043 1loss/SRqqt2/detection_loss‚2.4066099799596348 .loss/FBt2/detection_loss‚1.966431213542819 1loss/SRqqt4/assignment_loss‚1.632305624966438 0loss/FBt1/assignment_loss‚0.1302797708552882 /loss/FRt2/detection_loss‚1.9435023805466092 /loss/FRt3/detection_loss‚1.4535071426574644 /loss/FBt3/detection_loss‚0.4775688377179442 /loss/FRt1/assignment_loss‚3.243558448903701 2loss/SRqqt2/assignment_loss‚1.4454066180265868 1loss/SRqqt3/detection_loss‚0.8876751905890427 &loss/total_loss‚40.985846735210615 _step‚728 .loss/FBt4/detection_loss‚3.593235452969869 epoch‚7 /loss/SRqqt4/detection_loss‚2.46129727936708 2loss/SRqqt1/assignment_loss‚1.3389962702575657 1loss/SRqqt1/detection_loss‚2.1719399525059595 _runtime‚10435.306841002 /loss/FRt2/assignment_loss‚3.664856642484665 1loss/FBt2/assignment_loss‚0.04343469315164761 0loss/FBt3/assignment_loss‚0.0206913284434285 2loss/FBt4/assignment_loss‚0.023818444771071274؉F†‘î%‚(:† ’èçÔ²“¾‡ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚136.466' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.1165 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0••ï%‚(:ô ’èçÔ㙈# disk.nvme0n1.in‚329974.49609375 disk.dm-0.in‚531386.42578125 disk.sdb2.out‚1148.091796875 disk.md0.in‚ 10.97265625 disk.sdb.in‚5.40625 disk.sdb1.out‚0 disk.dm-0.out‚127672.1015625 disk.sda1.in‚0 disk.rbd0.in‚0.015625" disk.nvme1n1.in‚201411.9296875$ proc.memory.availableMB‚9949.125 network.sent‚ 2.3311601e+07 disk.sda.out‚1148.091796875 disk.sda1.out‚0 disk.rbd2.in‚0$ disk./.usageGB‚2974.083740234375 disk.md0p1.in‚ 10.97265625 disk.md0p1.out‚ 1106.41796875 disk.md0.out‚ 1106.41796875) proc.memory.percent‚18.50123405456543 cpu‚25.19756649132735 proc.cpu.threads‚17 disk.sda2.in‚ 5.56640625 disk.sdb2.in‚5.40625 disk.rbd2.out‚ 3.025390625 disk.sdb.out‚1148.091796875% memory_percent‚39.275360107421875" proc.memory.rssMB‚ 3031.2421875) disk./.usagePercent‚83.15662612770048 network.recv‚ 3.558059e+06 disk.sdb1.in‚0# disk.nvme0n1.out‚62296.79296875 disk.sda2.out‚1148.091796875 disk.rbd0.out‚ 3.087890625# disk.nvme1n1.out‚65375.30859375 disk.sda.in‚ 5.56640625ý~…—œð%‚(:‘ ¡èçÔÄ˳‡ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚119.978' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.994499999999995 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ËYo…ñ%‚(:ú ¡èçÔ­Ê»‡" disk.nvme0n1.out‚ 62296.8515625 disk.dm-0.in‚531386.42578125 disk.dm-0.out‚ 127741.140625 disk.sda1.out‚0 disk.sdb2.in‚5.40625 disk.sdb.in‚5.40625% memory_percent‚39.273786544799805" proc.memory.rssMB‚ 3031.2421875 disk.sdb1.in‚0 disk.rbd0.out‚ 3.087890625 disk.sdb1.out‚0 disk.sda.in‚ 5.56640625! disk.sda2.out‚1149.5126953125! disk.sdb2.out‚1149.5126953125 disk.rbd0.in‚0.015625 proc.cpu.threads‚17% disk./.usageGB‚2974.0838203430176 disk.sda.out‚1149.5126953125 disk.md0.in‚ 10.97265625 disk.md0.out‚ 1107.77734375 disk.sdb.out‚1149.5126953125) proc.memory.percent‚18.50123405456543 cpu‚25.19731429915918 disk.md0p1.in‚ 10.97265625 disk.md0p1.out‚ 1107.77734375 network.sent‚ 2.3418463e+07# disk.nvme0n1.in‚329974.49609375 disk.sda1.in‚0 disk.sda2.in‚ 5.56640625 disk.rbd2.in‚0" disk.nvme1n1.out‚ 65444.2890625) disk./.usagePercent‚83.15662836757166 disk.rbd2.out‚ 3.025390625 network.recv‚ 3.564503e+06" disk.nvme1n1.in‚201411.9296875( proc.memory.availableMB‚ 9949.3828125)5ç”’ò%‚(:‡ °èçÔ²…™† gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚118.751' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.68775 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿÉÀ¶ˆó%‚(:ý °èçÔý‹¬‡! disk.sdb2.out‚1150.9794921875 disk.rbd2.out‚ 3.025390625) proc.memory.percent‚18.50123405456543 disk.sdb.out‚1150.9794921875! disk.sda2.out‚1150.9794921875 disk.md0.out‚ 1109.17578125 proc.cpu.threads‚17 disk.sda.in‚ 5.56640625 disk.dm-0.in‚531386.42578125 network.sent‚ 2.3424675e+07 disk.sda.out‚1150.9794921875 disk.sdb2.in‚5.40625# disk.nvme1n1.out‚65605.19921875" disk.nvme0n1.out‚ 62302.9765625! disk.dm-0.out‚127908.17578125 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 3.087890625$ disk./.usageGB‚2974.089672088623# disk.nvme0n1.in‚329974.49609375 disk.sda1.in‚0" proc.memory.rssMB‚ 3031.2421875" disk.nvme1n1.in‚201411.9296875 disk.sda1.out‚0 disk.md0p1.out‚ 1109.17578125% memory_percent‚39.273810386657715 cpu‚25.19707808243956 disk.sdb.in‚5.40625 disk.sdb1.in‚0 disk.md0.in‚ 10.97265625 disk.rbd2.in‚0) proc.memory.availableMB‚ 9949.37890625 network.recv‚ 3.569357e+06) disk./.usagePercent‚83.15679198482793 disk.sdb1.out‚0 disk.sda2.in‚ 5.56640625 disk.md0p1.in‚ 10.97265625TúˆAô%‚(:… ¿èçÔ™—ˆ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚124.994' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.2485 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨Ò4‹õ%‚(:€  ¿èçÔ èÒ‰" disk.sda2.out‚1152.66064453125 disk.rbd2.out‚ 3.025390625 network.sent‚ 2.3432136e+07 network.recv‚ 3.573325e+06" disk.sdb2.out‚1152.66064453125 disk.rbd0.in‚0.015625" proc.memory.rssMB‚ 3031.2421875 cpu‚25.19685678036891) disk./.usagePercent‚83.14903744416108 disk.md0.in‚ 10.97265625 disk.sda.in‚ 5.56640625# disk.nvme0n1.in‚329974.49609375% disk./.usageGB‚2973.8123321533203 disk.rbd0.out‚ 3.087890625! disk.sdb.out‚1152.66064453125! disk.sda.out‚1152.66064453125" disk.nvme0n1.out‚ 62303.0390625 disk.md0p1.in‚ 10.97265625 disk.sdb2.in‚5.40625 disk.md0p1.out‚ 1110.7890625% memory_percent‚39.272284507751465) proc.memory.availableMB‚ 9949.62890625 proc.cpu.threads‚17# disk.nvme1n1.out‚65643.96484375) proc.memory.percent‚18.50123405456543 disk.sdb.in‚5.40625 disk.sdb1.in‚0 disk.sdb1.out‚0! disk.dm-0.out‚127947.00390625 disk.sda1.in‚0 disk.sda2.in‚ 5.56640625 disk.md0.out‚ 1110.7890625 disk.rbd2.in‚0 disk.dm-0.in‚531386.42578125" disk.nvme1n1.in‚201411.9296875 disk.sda1.out‚0‘]ãn4ö%/ _wandbruntime‚10488  _runtime‚104886'M›Ê÷%Ä ' lr-AdamW/pg2‚0.00016292262033842366 trainer/global_step‚8899 % _timestamp‚1.7884744349534328e+09 _runtime‚10488.138665372 _step‚729 ' lr-AdamW/pg1‚0.00016292262033842366ê«1éÏø%É _step‚729 ' lr-AdamW/pg1‚0.00016292262033842366 ' lr-AdamW/pg2‚0.00016292262033842366 trainer/global_step‚8899 % _timestamp‚1.7884744349534328e+09 _runtime‚10488.138665372ÙvŠ’4ù%/ _wandbruntime‚10488  _runtime‚10488UM&ú%êO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 169443451 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17640493002132 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚380612840019968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17690414645248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚362922425374720 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17690648313856 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚362922443377664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 242085259 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17690414645248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚402383276488192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚380613091691520 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚384393990166016 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚345305592542512 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚362946085544644 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 169443453 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282230784 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 411528710 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 _step‚730 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 169443453 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101298282 F@Device~ … £StatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17989592493056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104›-gõ'û%ïO ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚698 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚8899 _runtime‚10488.140001657 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚362946085544644 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 169443453 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282230784 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 411528710 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 _step‚730 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 169443453 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101298282 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17989592493056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 169443451 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17640493002132 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚380612840019968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17690414645248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚362922425374720 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17690648313856 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚362922443377664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 242085259 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17690414645248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚402383276488192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚380613091691520 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚384393990166016 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚345305592542512 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 169461728 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64317773 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 165627525 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚668 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17642451491326 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 411600049 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 411573677 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 % _timestamp‚1.7884744362066777e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚362961729414656 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 411600049 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 165626855 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 _runtime‚10489.392372658 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚380654341767680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 242111951 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚362961711411712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 169461728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64318441½@ˆ±'þ%úM YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17692378671104 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚362961711411712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17991304822784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚380654341767680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 242111951 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚362961711411712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 169461728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64318441 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 242111951 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 169461728 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64317773 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 165627525 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚668 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17642451491326 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 411600049 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 411573677 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 % _timestamp‚1.7884744362066777e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚362961729414656 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 411600049 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 165626855 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 _runtime‚10489.392372658 E¨……&‚(:ú ÝèçÔç¢Ú‡# disk.nvme0n1.in‚329974.49609375 disk.md0p1.in‚ 10.97265625 disk.sdb.out‚1155.8720703125 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sda2.in‚ 5.56640625 disk.md0.out‚ 1113.88671875! disk.dm-0.out‚128184.05078125 cpu‚25.19640157687727 disk.rbd2.in‚0 disk.sda1.out‚0$ memory_percent‚39.28043842315674 disk.rbd2.out‚ 3.025390625" disk.nvme1n1.in‚201411.9296875 disk.sda.in‚ 5.56640625 disk.md0p1.out‚ 1113.88671875! disk.sdb2.out‚1155.8720703125 disk.sda.out‚1155.8720703125) proc.memory.availableMB‚ 9948.29296875 proc.cpu.threads‚17 network.sent‚ 2.3544101e+07 network.recv‚ 3.5852e+06% disk./.usageGB‚2974.1208686828613 disk.sdb2.in‚5.40625 disk.rbd0.in‚0.015625 disk.sdb1.in‚0 disk.dm-0.in‚531386.42578125" proc.memory.rssMB‚ 3031.2421875! disk.sda2.out‚1155.8720703125 disk.md0.in‚ 10.97265625 disk.rbd0.out‚ 3.087890625! disk.nvme1n1.out‚ 65736.546875 disk.sdb.in‚5.40625# disk.nvme0n1.out‚62447.50390625) proc.memory.percent‚18.50123405456543) disk./.usagePercent‚83.15766425465961%ú1œ†&‚(:‘ ìèçÔ°ö±‡ gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³>d/ƒ‡&‚(:ø ìèçÔÙÖ‰ network.sent‚ 2.3550021e+07! disk.sdb2.out‚1157.5615234375 disk.sdb1.in‚0 disk.sda1.out‚0 disk.md0p1.out‚ 1115.515625$ memory_percent‚39.28043842315674) proc.memory.availableMB‚ 9948.29296875 cpu‚25.19618371585744 disk.sda.out‚1157.5615234375 proc.cpu.threads‚17 disk.dm-0.in‚531386.42578125 disk.sda2.in‚ 5.56640625 disk.sdb2.in‚5.40625 disk.md0.out‚ 1115.515625 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.rbd0.out‚ 3.087890625 disk.sda.in‚ 5.56640625! disk.sda2.out‚1157.5615234375" disk.nvme1n1.in‚201411.9296875 disk.md0.in‚ 10.97265625# disk.nvme0n1.in‚329974.49609375) proc.memory.percent‚18.50123405456543 disk.rbd2.out‚ 3.025390625 disk.sdb.out‚1157.5615234375" proc.memory.rssMB‚ 3031.2421875 disk.sda1.in‚0# disk.nvme1n1.out‚65782.70703125 network.recv‚ 3.589186e+06$ disk./.usageGB‚2974.339282989502# disk.nvme0n1.out‚62519.64453125 disk.md0p1.in‚ 10.97265625) disk./.usagePercent‚83.16377121008678 disk.rbd0.in‚0.015625 disk.sdb.in‚5.40625 disk.dm-0.out‚128302.3515625Œ‚Z ‘ˆ&‚(:† ûèçÔá‚Š gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.966' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.9915 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒI߆‰&‚(:û ûèçÔŪ¥% disk./.usageGB‚2974.5886726379395# disk.nvme0n1.out‚62591.69921875) proc.memory.percent‚18.50123405456543 disk.rbd2.in‚0 disk.sdb.out‚1159.6279296875 disk.sdb1.out‚0 disk.sda.out‚1159.6279296875 disk.md0.out‚ 1117.51171875 disk.sdb.in‚5.40625 disk.dm-0.out‚ 128466.296875# memory_percent‚39.2789363861084" proc.memory.rssMB‚ 3031.2421875 disk.rbd2.out‚ 3.025390625" disk.nvme1n1.in‚201411.9296875# disk.nvme0n1.in‚329974.49609375( proc.memory.availableMB‚ 9948.5390625 proc.cpu.threads‚17 network.sent‚ 2.3555279e+07! disk.sdb2.out‚1159.6279296875 disk.sdb1.in‚0 disk.sda.in‚ 5.56640625 disk.sda1.out‚0 disk.md0.in‚ 10.97265625 disk.rbd0.in‚0.015625 disk.sda2.in‚ 5.56640625 network.recv‚ 3.592774e+06) disk./.usagePercent‚83.17074424903477 disk.sda1.in‚0! disk.sda2.out‚1159.6279296875 disk.md0p1.in‚ 10.97265625 disk.md0p1.out‚ 1117.51171875 cpu‚25.195958447978207 disk.sdb2.in‚5.40625 disk.rbd0.out‚ 3.087890625# disk.nvme1n1.out‚65874.59765625 disk.dm-0.in‚531386.42578125aÝ&4Š&/ _wandbruntime‚10547  _runtime‚10547\Wý:É‹&à _runtime‚10547.244084721 _step‚733 ' lr-AdamW/pg1‚0.00015820943647934727 ' lr-AdamW/pg2‚0.00015820943647934727 trainer/global_step‚8949 $ _timestamp‚1.788474494058894e+09(=yÎŒ&È ' lr-AdamW/pg1‚0.00015820943647934727 ' lr-AdamW/pg2‚0.00015820943647934727 trainer/global_step‚8949 $ _timestamp‚1.788474494058894e+09 _runtime‚10547.244084721 _step‚733Ý[ 1=4&/ _wandbruntime‚10547  _runtime‚10547”|q9ñ'Ž&ëO XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17788630351872 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18089210864128 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 170357203 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 243449555 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17788864020480 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚382675609247744 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚382675357576192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101838382 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚404567375592448 _runtime‚10547.245758059 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18089503580672 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 170357201 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 166495879 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17788864020480 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚382675357576192 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚347182093374352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 % _timestamp‚1.7884744940603826e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64657497 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚8949 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚364886745227264 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64656794 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17738431042640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚364920541456352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚364886727224320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚364886727224320 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚347182093374352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 % _timestamp‚1.7884744940603826e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64657497 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚8949 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚364886745227264 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64656794 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17738431042640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚364920541456352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚364886727224320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚364886727224320 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101838382 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚404567375592448 _runtime‚10547.245758059 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18089503580672 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 170357201 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 166495879 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17788864020480 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚382675357576192 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 413825379 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 166512888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_p2%1kãool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275926016 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 % _timestamp‚1.7884744951806664e+09 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17740389531834 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 413851751 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚683 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17790828059648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚364959784282318 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚347219640771136 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101849182 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚382716859323904 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64663706 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17740160550542 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 413851751 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚404611063920640 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚364960030302970 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚382716607639040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 170375476 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17790594377728 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _step‚735 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚382716859323904 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64664387 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17790594377728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 243449903 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚8949 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372zÕ?'’&úM PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 413825379 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 166512888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275926016 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 % _timestamp‚1.7884744951806664e+09 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17740389531834 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 413851751 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚683 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17790828059648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚364959784282318 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚347219640771136 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101849182 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚382716859323904 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64663706 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17740160550542 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 413851751 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚404611063920640 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚364960030302970 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚382716607639040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 170375476 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17790594377728 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _step‚735 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚382716859323904 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64664387 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17790594377728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 243449903 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚8949 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 413825379 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 243476273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚364926031264256 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚364926013261312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 243449903 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚386519848698368 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚364926013261312 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17790828059648 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18091215222272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582ßk>8Â4“&/ _wandbruntime‚10548  _runtime‚10548-|Ó^”&Ø 1loss/FBt4/assignment_loss‚0.33612489640555526 .loss/FBt1/detection_loss‚3.907207977771759 0loss/FBt3/detection_loss‚0.44367978745140135 1loss/FBt3/assignment_loss‚0.06799602219727449 -loss/FRt4/detection_loss‚3.24009626067203 % _timestamp‚1.7884744951844087e+09 0loss/FBt1/assignment_loss‚0.3133211412813125 /loss/FRt1/detection_loss‚2.8370051542917887 1loss/SRqqt1/detection_loss‚2.2728776114327567 %loss/total_loss‚43.11730224703858 0loss/FRt4/assignment_loss‚2.6009088625078616 /loss/FRt2/detection_loss‚2.2149217514731725 .loss/FBt2/detection_loss‚2.028052915545071 trainer/global_step‚8949 1loss/SRqqt2/detection_loss‚2.1891254674304617 0loss/FRt1/assignment_loss‚2.7461718599001568 0loss/FRt2/assignment_loss‚3.8111839200320996 2loss/SRqqt2/assignment_loss‚1.6744652381449034 .loss/FRt3/detection_loss‚1.385133561718723 /loss/FBt4/detection_loss‚2.1319422741731007 epoch‚7 2loss/SRqqt1/assignment_loss‚1.1612508646079471 2loss/SRqqt4/assignment_loss‚1.7214605584740639 1loss/FBt2/assignment_loss‚0.08021493906005492 1loss/SRqqt3/detection_loss‚0.8206564859239334 _step‚736 /loss/FRt3/assignment_loss‚1.262532920077227 2loss/SRqqt3/assignment_loss‚0.8499938090066981 1loss/SRqqt4/detection_loss‚3.0209779674592228 _runtime‚10548.369201241¿÷É‹c•&Ý epoch‚7 2loss/SRqqt1/assignment_loss‚1.1612508646079471 2loss/SRqqt4/assignment_loss‚1.7214605584740639 1loss/FBt2/assignment_loss‚0.08021493906005492 1loss/SRqqt3/detection_loss‚0.8206564859239334 _step‚736 /loss/FRt3/assignment_loss‚1.262532920077227 2loss/SRqqt3/assignment_loss‚0.8499938090066981 1loss/SRqqt4/detection_loss‚3.0209779674592228 _runtime‚10548.369201241 1loss/FBt4/assignment_loss‚0.33612489640555526 .loss/FBt1/detection_loss‚3.907207977771759 0loss/FBt3/detection_loss‚0.44367978745140135 1loss/FBt3/assignment_loss‚0.06799602219727449 -loss/FRt4/detection_loss‚3.24009626067203 % _timestamp‚1.7884744951844087e+09 0loss/FBt1/assignment_loss‚0.3133211412813125 /loss/FRt1/detection_loss‚2.8370051542917887 1loss/SRqqt1/detection_loss‚2.2728776114327567 %loss/total_loss‚43.11730224703858 0loss/FRt4/assignment_loss‚2.6009088625078616 /loss/FRt2/detection_loss‚2.2149217514731725 .loss/FBt2/detection_loss‚2.028052915545071 trainer/global_step‚8949 1loss/SRqqt2/detection_loss‚2.1891254674304617 0loss/FRt1/assignment_loss‚2.7461718599001568 0loss/FRt2/assignment_loss‚3.8111839200320996 2loss/SRqqt2/assignment_loss‚1.6744652381449034 .loss/FRt3/detection_loss‚1.385133561718723 /loss/FBt4/detection_loss‚2.1319422741731007àXjñ’–&‚(:‡ ŠéçÔ‰óȆ gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚139.251' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.81275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^aSx—&‚(:í ŠéçÔ™¤õ† disk.rbd2.in‚0 disk.sda2.out‚ 1161.29296875 disk.md0p1.out‚ 1119.109375) proc.memory.availableMB‚ 9948.25390625) proc.memory.percent‚18.50123405456543 disk.nvme1n1.out‚ 65966.625 disk.sdb1.in‚0 disk.sda.out‚ 1161.29296875 disk.sdb2.out‚ 1161.29296875" proc.memory.rssMB‚ 3031.2421875 disk.sda2.in‚ 5.56640625 disk.sdb2.in‚5.40625 network.recv‚ 3.597432e+06" disk.nvme1n1.in‚201411.9296875 disk.sda.in‚ 5.56640625# disk.nvme0n1.out‚62663.91015625 disk.rbd0.out‚ 3.087890625# disk.nvme0n1.in‚329974.49609375$ memory_percent‚39.28067684173584 network.sent‚ 2.3561653e+07! disk.dm-0.out‚128630.53515625 disk.sda1.in‚0 disk.md0p1.in‚ 10.97265625 proc.cpu.threads‚17 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 3.025390625 disk.sdb.in‚5.40625 disk.dm-0.in‚531386.42578125 cpu‚25.19575981037319% disk./.usageGB‚2974.5897789001465 disk.sdb.out‚ 1161.29296875 disk.sda1.out‚0 disk.md0.in‚ 10.97265625 disk.md0.out‚ 1119.109375 disk.sdb1.out‚0) disk./.usagePercent‚83.17077518058909òêØ&‚(:… ™éçÔ¸¬”† gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚139.868' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.967 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»äå%„™&‚(:ù ™éçÔ¨ŠÕ‡ disk.dm-0.in‚531386.42578125 network.recv‚ 3.60466e+06" disk.nvme1n1.in‚201411.9296875! disk.sdb.out‚1162.59130859375# disk.nvme0n1.in‚329974.49609375 disk.md0.out‚ 1120.34375 disk.sdb1.out‚0 disk.sda1.in‚0) proc.memory.percent‚18.50123405456543" disk.sda2.out‚1162.59130859375 disk.md0p1.in‚ 10.97265625 disk.sdb2.in‚5.40625" disk.sdb2.out‚1162.59130859375 disk.rbd0.in‚0.015625$ memory_percent‚39.28067684173584 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.rbd2.out‚ 3.025390625 disk.sdb.in‚5.40625 network.sent‚ 2.3669289e+07 disk.md0p1.out‚ 1120.34375 disk.rbd0.out‚ 3.087890625 disk.sdb1.in‚0# disk.nvme0n1.out‚62735.98828125 disk.dm-0.out‚ 128840.40625) disk./.usagePercent‚83.16303023937013# disk.nvme1n1.out‚66104.41796875" proc.memory.rssMB‚ 3031.2421875% disk./.usageGB‚2974.3127822875977 disk.sda.in‚ 5.56640625 disk.sda1.out‚0) proc.memory.availableMB‚ 9948.25390625 cpu‚25.195526469442267 disk.sda2.in‚ 5.56640625 disk.md0.in‚ 10.97265625! disk.sda.out‚1162.59130859375Öwx†‘š&‚(:† ¨éçÔ²Õ‡ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.357' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.83925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ãü´¤›&‚(:ô ¨éçÔ󜄉) disk./.usagePercent‚83.17594522990208 disk.sda.out‚1164.0283203125" disk.nvme0n1.out‚ 62808.1640625 disk.dm-0.out‚128951.9140625 disk.sdb2.in‚5.40625 disk.rbd0.out‚ 3.087890625 disk.sdb1.in‚0 disk.sdb1.out‚0) proc.memory.availableMB‚ 9948.25390625! disk.sdb2.out‚1164.0283203125 disk.rbd2.in‚0 disk.md0p1.out‚ 1121.7109375 disk.md0.in‚ 10.97265625 disk.md0.out‚ 1121.7109375 disk.rbd0.in‚0.015625$ disk./.usageGB‚2974.774684906006 disk.sda.in‚ 5.56640625# disk.nvme0n1.in‚329974.49609375 disk.dm-0.in‚531386.42578125 disk.md0p1.in‚ 10.97265625$ memory_percent‚39.28067684173584 disk.sda2.in‚ 5.56640625! disk.sda2.out‚1164.0283203125 disk.rbd2.out‚ 3.025390625 disk.sdb.in‚5.40625 proc.cpu.threads‚17 network.sent‚ 2.3677574e+07 network.recv‚ 3.608878e+06 disk.nvme1n1.out‚66143.75) proc.memory.percent‚18.50123405456543" proc.memory.rssMB‚ 3031.2421875 cpu‚25.195314241884947" disk.nvme1n1.in‚201411.9296875 disk.sdb.out‚1164.0283203125 disk.sda1.in‚0 disk.sda1.out‚0ý ˆ1œœ&‚(:‘ ·éçÔ£æׇ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚110.681' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚27.670250000000003 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƳF†t&‚(:é ·éçԳɉ disk.nvme0n1.out‚ 62880.3125 cpu‚25.195083210126082) disk./.usagePercent‚83.17771398150626 disk.md0.in‚ 10.97265625" disk.nvme1n1.out‚ 66210.6640625 disk.sdb.out‚ 1166.046875$ memory_percent‚39.27915096282959" proc.memory.rssMB‚ 3031.2421875 proc.cpu.threads‚17 disk.rbd0.in‚0.015625" disk.nvme1n1.in‚201411.9296875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚ 1166.046875 disk.sdb2.in‚5.40625) proc.memory.percent‚18.50123405456543 disk.rbd0.out‚ 3.087890625 disk.rbd2.in‚0 disk.sdb1.in‚0 disk.sda.out‚ 1166.046875 disk.dm-0.in‚531386.42578125 disk.dm-0.out‚129090.9765625 disk.sda2.in‚ 5.56640625 network.recv‚ 3.612448e+06 disk.sdb1.out‚0 disk.md0p1.in‚ 10.97265625 network.sent‚ 2.3682882e+07 disk.md0.out‚ 1123.66796875 disk.rbd2.out‚ 3.025390625 disk.sdb2.out‚ 1166.046875# disk.nvme0n1.in‚329974.49609375 disk.md0p1.out‚ 1123.66796875) proc.memory.availableMB‚ 9948.50390625% disk./.usageGB‚2974.8379440307617 disk.sdb.in‚5.40625 disk.sda.in‚ 5.56640625<|D 4ž&/ _wandbruntime‚10606  _runtime‚10606ÓTÆŸ&À & lr-AdamW/pg1‚0.0001535491893645425 & lr-AdamW/pg2‚0.0001535491893645425 trainer/global_step‚8999 # _timestamp‚1.78847455311207e+09 _runtime‚10606.297129093 _step‚737λ¼EË &Å & lr-AdamW/pg1‚0.0001535491893645425 & lr-AdamW/pg2‚0.0001535491893645425 trainer/global_step‚8999 # _timestamp‚1.78847455311207e+09 _runtime‚10606.297129093 _step‚737áÖ-1Ë4¡&/ _wandbruntime‚10606  _runtime‚10606<›”s¢&ëO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚688 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 244761163 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚384737875132416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65008378 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 trainer/global_step‚8999 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 171270951 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚366851029073920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚366894997368060 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚366851047076864 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 D:DeviceStatsMorðÅz~ nitor.on_train_batch_start/active.all.current‚26346 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚384737875132416 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18189016632832 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 171270953 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 244787507Aâü4ö'£&ðO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 171270953 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 244787507 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚688 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 244761163 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚384737875132416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65008378 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 trainer/global_step‚8999 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 171270951 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚366851029073920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚366894997368060 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚366851047076864 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚384737875132416 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18189016632832 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0⃯Ëw4¤&/ _wandbruntime‚10607  _runtime‚10607,á‹ú&¥&ôM L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 167404235 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17838098591050 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17889043766272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17889043766272 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17888810084352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65014953 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚366890315110912 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚366890333113856 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 102389284 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚666 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚388604047200768 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚349096141602976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 244787855 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 244814225 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18191301197312 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚8999 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚384779376880128 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65015619 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 416103453 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 171289228 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚406795055694848 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 167404903 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚388604071140864 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17838327572342 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 _step‚739 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚668 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚406795372338176 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17888810084352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚366890333113856 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 416077081 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 416103453 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚384779376880128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 102389282 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 _runtime‚10607.432643435 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 416077081 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚349096158642336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 167404235 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17838098591050 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17889043766272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17889043766272 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17888810084352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65014953 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚366890315110912 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚366890333113856 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 102389284 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚666 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚388604047200768 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚349096141602976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 244787855 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 244814225 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18191301197312 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚8999 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚384779376880128 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65015619 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 416103453 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 171289228 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚406795055694848 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 167404903 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚388604071140864 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17838327572342 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 _step‚739 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚668 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚406795372338176 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17888810084352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚366890333113856 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 416077081 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 416103453 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚384779376880128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 102389282 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 _runtime‚10607.432643435 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 416077081 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚349096158642336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚408935707623424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚390646561733120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 418283816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚368815348926464 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚368815330923520 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚368869207259324 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 168274833 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65355549 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18289121950208 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 _step‚742 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 172184701 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚386800644360192 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 172184703 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 168274129 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17934307123656 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚704 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 278036480 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 # _timestamp‚1.78847461226062e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 102918580 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 172184703 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚368815330923520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17985295433728 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301976576 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚390646585673216 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚386800392688640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚350935129116752 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17985061765120 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 102918582 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 _runtime‚10665.446131546 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚386800392688640 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 246099115 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 418310162 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 246125459 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18288843913728 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 172184701 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚368869453279768 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65356251 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚350935146156112 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17934078142572 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚170393606úÁÍô'·&îO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65356251 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚350935146156112 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17934078142572 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 246099115 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚408935707623424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚390646561733120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 418283816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚368815348926464 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚368815330923520 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚368869207259324 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 168274833 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65355549 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18289121950208 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 _step‚742 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 172184701 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚386800644360192 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 172184703 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 168274129 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17934307123656 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚704 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 278036480 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 # _timestamp‚1.78847461226062e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 102918580 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 172184703 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚368815330923520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17985295433728 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301976576 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚390646585673216 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚386800392688640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚350935129116752 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17985061765120 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 102918582 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 _runtime‚10665.446131546 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚386800392688640 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 246099115 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 418310162 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 246125459 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18288843913728 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 172184701 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚368869453279768 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309æe¡×4¸&/ _wandbruntime‚10666  _runtime‚10666‹x ™X ¹&õM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚657 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 246152177 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚10666.585570387 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65362815 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 102929384 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65363472 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚350972659474176 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17987025790976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _step‚743 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚368908942126386 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17987259472896 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 418355155 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17987259472896 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 246152177 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18290836540928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚368854616960512 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17936036631558 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 172202976 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 168292197 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚386841642751488 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 418328783 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚390688269643264 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 172202978 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 102929382 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚408979388401664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273Ùq©½'º&úM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚368854616960512 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17936036631558 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 172202976 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 168292197 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚386841642751488 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 418328783 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚390688269643264 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 172202978 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 102929382 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚408979388401664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚657 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 246152177 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚10666.585570387 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65362815 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 102929384 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65363472 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚350972659474176 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17987025790976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _step‚743 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚368908942126386 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17987259472896 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 418355155 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17987259472896 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 246152177 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18290836540928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582瑳 R4»&/ _wandbruntime‚10666  _runtime‚10666æ—°’]¼&× 1loss/FBt3/assignment_loss‚0.09906857522568571 %loss/total_loss‚42.41808491681717 1loss/FBt4/assignment_loss‚0.04696843530981596 0loss/FRt1/assignment_loss‚3.4025000240653753 /loss/FRt2/assignment_loss‚3.260891238848368 /loss/FRt4/assignment_loss‚3.085398777535087 2loss/SRqqt1/assignment_loss‚1.6203012101662655 1loss/SRqqt3/assignment_loss‚1.024866510085052 /loss/FRt3/assignment_loss‚1.251353625456492 0loss/FBt2/assignment_loss‚0.2461526280530901 trainer/global_step‚9049 _runtime‚10666.586993553 2loss/SRqqt4/assignment_loss‚1.6696621586258213 1loss/SRqqt2/detection_loss‚2.2855280073913367 epoch‚7 _step‚744 2loss/SRqqt2/assignment_loss‚1.4584168699142095 0loss/FBt1/assignment_loss‚0.3825194333595153 /loss/FRt1/detection_loss‚2.6560337767004967 /loss/FRt3/detection_loss‚1.4240492532153923 /loss/FRt4/detection_loss‚2.8011550134734104 1loss/SRqqt4/detection_loss‚2.5259079498549304 /loss/FBt1/detection_loss‚1.9564803080125288 /loss/FBt2/detection_loss‚1.6607101697188158 0loss/FBt3/detection_loss‚0.42022482970390806 .loss/FBt4/detection_loss‚4.467754375934601 .loss/FRt2/detection_loss‚2.038562418965416 1loss/SRqqt1/detection_loss‚1.7776032102604706 1loss/SRqqt3/detection_loss‚0.8559761169410888 % _timestamp‚1.7884746134020927e+09K¯L‚b½&Ü 0loss/FBt1/assignment_loss‚0.3825194333595153 /loss/FRt1/detection_loss‚2.6560337767004967 /loss/FRt3/detection_loss‚1.4240492532153923 /loss/FRt4/detection_loss‚2.8011550134734104 1loss/SRqqt4/detection_loss‚2.5259079498549304 /loss/FBt1/detection_loss‚1.9564803080125288 /loss/FBt2/detection_loss‚1.6607101697188158 0loss/FBt3/detection_loss‚0.42022482970390806 .loss/FBt4/detection_loss‚4.467754375934601 .loss/FRt2/detection_loss‚2.038562418965416 1loss/SRqqt1/detection_loss‚1.7776032102604706 1loss/SRqqt3/detection_loss‚0.8559761169410888 % _timestamp‚1.7884746134020927e+09 1loss/FBt3/assignment_loss‚0.09906857522568571 %loss/total_loss‚42.41808491681717 1loss/FBt4/assignment_loss‚0.04696843530981596 0loss/FRt1/assignment_loss‚3.4025000240653753 /loss/FRt2/assignment_loss‚3.260891238848368 /loss/FRt4/assignment_loss‚3.085398777535087 2loss/SRqqt1/assignment_loss‚1.6203012101662655 1loss/SRqqt3/assignment_loss‚1.024866510085052 /loss/FRt3/assignment_loss‚1.251353625456492 0loss/FBt2/assignment_loss‚0.2461526280530901 trainer/global_step‚9049 _runtime‚10666.586993553 2loss/SRqqt4/assignment_loss‚1.6696621586258213 1loss/SRqqt2/detection_loss‚2.2855280073913367 epoch‚7 _step‚744 2loss/SRqqt2/assignment_loss‚1.4584168699142095èúT€q‘¾&‚(:† ‚êçÔÅá… gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚124.994' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.2485 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š»Š{w¿&‚(:ì ‚êçÔ÷Œâˆ disk.rbd2.out‚ 3.025390625 disk.sdb.out‚ 1174.3359375& proc.memory.availableMB‚ 9947.90625 disk.sda1.in‚0 disk.md0.out‚ 1131.68359375" disk.nvme1n1.in‚201411.9296875# disk.nvme0n1.in‚329974.49609375) proc.memory.percent‚18.50123405456543 network.sent‚ 2.3812436e+07 disk.sdb2.in‚5.40625 disk.sda.in‚ 5.56640625% disk./.usageGB‚2974.8127365112305 disk.sdb1.out‚0% memory_percent‚39.282798767089844 disk.sdb2.out‚ 1174.3359375 disk.sda1.out‚0 disk.rbd0.in‚0.015625# disk.nvme1n1.out‚66626.78515625 disk.sdb.in‚5.40625& disk./.usagePercent‚83.17700916871! disk.dm-0.out‚129867.74609375 disk.md0p1.in‚ 10.97265625 disk.md0p1.out‚ 1131.68359375 disk.md0.in‚ 10.97265625 disk.sdb1.in‚0 disk.sda.out‚ 1174.3359375" disk.nvme0n1.out‚ 63240.9609375 disk.sda2.out‚ 1174.3359375 disk.rbd0.out‚ 3.087890625 disk.rbd2.in‚0" proc.memory.rssMB‚ 3031.2421875 cpu‚25.19402475595086 proc.cpu.threads‚17 network.recv‚ 3.636061e+06 disk.dm-0.in‚531386.42578125 disk.sda2.in‚ 5.56640625 <²’À&‚(:‡ ‘êçÔÐè‡ gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚137.083' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.27075 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸L~xÁ&‚(:í ‘êçÔñ§‰ disk.sda2.in‚ 5.56640625 disk.sdb2.out‚ 1176.2421875 disk.sdb1.out‚0# disk.nvme0n1.out‚63313.00390625 disk.dm-0.in‚531386.42578125" proc.memory.rssMB‚ 3031.2421875) proc.memory.percent‚18.50123405456543 proc.cpu.threads‚17 disk.sda.out‚ 1176.2421875# disk.nvme0n1.in‚329974.49609375 disk.md0.in‚ 10.97265625" disk.nvme1n1.in‚201411.9296875 disk.sdb.in‚5.40625 disk.sdb.out‚ 1176.2421875 disk.sdb1.in‚0 disk.md0p1.out‚ 1133.5390625 disk.rbd0.out‚ 3.087890625 disk.rbd2.in‚0 disk.rbd2.out‚ 3.025390625% memory_percent‚39.279747009277344) disk./.usagePercent‚83.17700362236234 disk.sdb2.in‚5.40625 disk.sda.in‚ 5.56640625 network.recv‚ 3.643235e+06 disk.dm-0.out‚ 129978.515625 network.sent‚ 2.3920069e+07 disk.sda2.out‚ 1176.2421875# disk.nvme1n1.out‚66665.51171875% disk./.usageGB‚2974.8125381469727 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0p1.in‚ 10.97265625 disk.md0.out‚ 1133.5390625 disk.rbd0.in‚0.015625& proc.memory.availableMB‚ 9948.40625 cpu‚25.193820870133955€›+4’Â&‚(:‡  êçÔÒÁ⇠gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚100.759' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.18975 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l¦a„Ã&‚(:ù  êçÔÊäˆ disk.rbd0.in‚0.015625 cpu‚25.19359925432046) disk./.usagePercent‚83.17771419482732% disk./.usageGB‚2974.8379516601562 disk.sdb1.in‚0 disk.sda1.out‚0 disk.md0.out‚ 1135.08203125% memory_percent‚39.281272888183594 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sdb.in‚5.40625 disk.sda.out‚1177.8427734375 disk.sda1.in‚0! disk.sda2.out‚1177.8427734375 disk.sdb2.in‚5.40625) proc.memory.percent‚18.50123405456543 disk.rbd2.out‚ 3.025390625 disk.md0p1.in‚ 10.97265625& proc.memory.availableMB‚ 9948.15625 network.recv‚ 3.646638e+06# disk.nvme1n1.out‚66732.99609375 disk.md0p1.out‚ 1135.08203125" disk.nvme1n1.in‚201411.9296875 disk.dm-0.in‚531386.42578125 disk.md0.in‚ 10.97265625" proc.memory.rssMB‚ 3031.2421875 network.sent‚ 2.392524e+07 disk.sdb1.out‚0 disk.sda.in‚ 5.56640625# disk.nvme0n1.out‚63385.11328125 disk.dm-0.out‚ 130118.109375 disk.sda2.in‚ 5.56640625! disk.sdb2.out‚1177.8427734375 disk.rbd0.out‚ 3.087890625 disk.sdb.out‚1177.8427734375# disk.nvme0n1.in‚329974.49609375!S3o4Ä&/ _wandbruntime‚10724  _runtime‚10724,Å…CÊÅ&Ä _step‚745 ' lr-AdamW/pg1‚0.00014439137888746983 ' lr-AdamW/pg2‚0.00014439137888746983 trainer/global_step‚9099 % _timestamp‚1.7884746714268436e+09 _runtime‚10724.611753413Ó LËÏÆ&É % _timestamp‚1.7884746714268436e+09 _runtime‚10724.611753413 _step‚745 ' lr-AdamW/pg1‚0.00014439137888746983 ' lr-AdamW/pg2‚0.00014439137888746983 trainer/global_step‚9099éÕAê4Ç&/ _wandbruntime‚10724  _runtime‚10724ùþMÑÈ&ëO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65703441 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚388862910244864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65704122 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 169162804 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 103458682 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18388713964544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 247463411 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚411119790856704 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚388862910244864 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 247437067 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚388863161916416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚352811646987952 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚370779650776064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚370779650776064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18032245164164 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18083511140352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 420561864 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚392730760235520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚352811664027312 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18032016183080 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚388863161916416 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 trainer/global_step‚9099 YEDevóyô iceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚370843909191476 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚746 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚411119474200064 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18389006681088 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚392730784175616 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚370843663171032 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚370779632773120 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18083277471744 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18083511140352 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 173098453û]ö'É&ðO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65703441 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚388862910244864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65704122 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 169162804 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 103458682 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18388713964544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 247463411 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚411119790856704 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚388862910244864 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 247437067 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚388863161916416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚352811646987952 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚370779650776064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚370779650776064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18032245164164 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18083511140352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 420561864 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚392730760235520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚352811664027312 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18032016183080 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚388863161916416 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 trainer/global_step‚9099 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚370843909191476 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚746 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚411119474200064 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18389006681088 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚392730784175616 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚370843663171032 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚370779632773120 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18083277471744 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18083511140352 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 173098453êOàñÊ&‚(:… ¯êçÔ„Ìý… gpu.0.gpu‚39.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚143.872' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.968 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÂsÀƒzË&‚(:ï ¯êçÔºŽ‡‡ disk.rbd2.in‚0 disk.sdb.out‚ 1179.29296875 disk.dm-0.in‚531386.42578125 disk.sdb2.out‚ 1179.29296875 disk.rbd0.in‚0.015625" disk.nvme1n1.in‚201411.9296875 disk.sdb1.in‚0 disk.sda.in‚ 5.56640625# disk.nvme0n1.in‚329974.49609375" proc.memory.rssMB‚ 3031.2421875 disk.sda2.out‚ 1179.29296875 disk.rbd0.out‚ 3.087890625 disk.rbd2.out‚ 3.025390625 disk.sda.out‚ 1179.29296875 network.sent‚ 2.3930577e+07 network.recv‚ 3.650261e+06 disk.sda1.out‚0$ memory_percent‚39.28117752075195 disk.sda2.in‚ 5.56640625 disk.sdb2.in‚5.40625 disk.md0.out‚ 1136.4765625' proc.memory.availableMB‚ 9948.171875 cpu‚25.193384197971927 disk.md0p1.in‚ 10.97265625 disk.md0.in‚ 10.97265625) proc.memory.percent‚18.50123405456543) disk./.usagePercent‚83.17951654449911 disk.sda1.in‚0 disk.md0p1.out‚ 1136.4765625 proc.cpu.threads‚17$ disk./.usageGB‚2974.902412414551! disk.dm-0.out‚130287.12109375# disk.nvme1n1.out‚66757.96484375 disk.sdb.in‚5.40625 disk.sdb1.out‚0 disk.nvme0n1.out‚ 63529.15625ôc¬`4Ì&/ _wandbruntime‚10725  _runtime‚10725äÆ_9û&Í&õM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚370818918810112 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18085475179520 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 % _timestamp‚1.7884746723811405e+09 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 169180759 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚392772444205568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 420580485 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 103469484 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚388904160307712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 420606857 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 420580485 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚388904411992576 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚388904411992576 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 173116726 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚370883152017442 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18085241497600 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 _runtime‚10725.566597874 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚686 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚392772468145664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18390711373824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 247490129 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 247490129 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚352849177345376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 103469482 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚370818918810112 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚9099 _step‚747 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 169180071 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18085475179520 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 420606857 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65711275 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18390999882752 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18085241497600 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚411163155579392 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚352849194384736 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 247463759 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 169180071 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18085475179520 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 420606857 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65711275 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18390999882752 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18085241497600 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚411163155579392 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚352849194384736 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 247463759 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 420580485 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 103469484 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚388904160307712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 420606857 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 420580485 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚388904411992576 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚388904411992576 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 173116726 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚370883152017442 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18085241497600 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 _runtime‚10725.566597874 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚686 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 REDeviceStatsMonʼn¾m€itor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚392772468145664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18390711373824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 247490129 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 247490129 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚352849177345376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 103469482ëqžå4Ï&/ _wandbruntime‚10725  _runtime‚10725–KFaÐ&Û /loss/FBt3/detection_loss‚0.5871228391993536 _step‚748 0loss/FRt1/assignment_loss‚2.6510037978490195 2loss/SRqqt4/assignment_loss‚1.4299181845039128 1loss/FBt2/assignment_loss‚0.23415995214600116 .loss/FRt2/detection_loss‚2.100288069314653 /loss/FRt3/detection_loss‚1.8242204052512534 _runtime‚10725.568154811 /loss/FBt4/detection_loss‚2.7430318742990494 &loss/total_loss‚43.249333197562876 /loss/FRt4/assignment_loss‚3.503075295024448 1loss/SRqqt2/detection_loss‚2.4762722931124945 1loss/SRqqt3/detection_loss‚0.9224847248237994 trainer/global_step‚9099 0loss/FRt2/assignment_loss‚3.1221735005577407 0loss/FRt3/assignment_loss‚1.3535963203758001 2loss/SRqqt2/assignment_loss‚1.5091549248406382 2loss/SRqqt3/assignment_loss‚0.9073447566634665 /loss/FRt4/detection_loss‚2.9647593145017272 .loss/FBt2/detection_loss‚2.916083323104041 $ _timestamp‚1.788474672383463e+09 1loss/FBt4/assignment_loss‚0.11034200022307417 /loss/FRt1/detection_loss‚2.9051782935857773 1loss/SRqqt1/detection_loss‚1.9392161050508188 /loss/FBt1/detection_loss‚2.6006512003285542 1loss/FBt3/assignment_loss‚0.04665425865864828 1loss/SRqqt4/detection_loss‚2.3280761105673653 epoch‚7 2loss/SRqqt1/assignment_loss‚1.9568474991030471 1loss/FBt1/assignment_loss‚0.11767815447819073ÂÏ[¸fÑ&à 2loss/SRqqt1/assignment_loss‚1.9568474991030471 1loss/FBt1/assignment_loss‚0.11767815447819073 /loss/FBt3/detection_loss‚0.5871228391993536 _step‚748 0loss/FRt1/assignment_loss‚2.6510037978490195 2loss/SRqqt4/assignment_loss‚1.4299181845039128 1loss/FBt2/assignment_loss‚0.23415995214600116 .loss/FRt2/detection_loss‚2.100288069314653 /loss/FRt3/detection_loss‚1.8242204052512534 _runtime‚10725.568154811 /loss/FBt4/detection_loss‚2.7430318742990494 &loss/total_loss‚43.249333197562876 /loss/FRt4/assignment_loss‚3.503075295024448 1loss/SRqqt2/detection_loss‚2.4762722931124945 1loss/SRqqt3/detection_loss‚0.9224847248237994 trainer/global_step‚9099 0loss/FRt2/assignment_loss‚3.1221735005577407 0loss/FRt3/assignment_loss‚1.3535963203758001 2loss/SRqqt2/assignment_loss‚1.5091549248406382 2loss/SRqqt3/assignment_loss‚0.9073447566634665 /loss/FRt4/detection_loss‚2.9647593145017272 .loss/FBt2/detection_loss‚2.916083323104041 $ _timestamp‚1.788474672383463e+09 1loss/FBt4/assignment_loss‚0.11034200022307417 /loss/FRt1/detection_loss‚2.9051782935857773 1loss/SRqqt1/detection_loss‚1.9392161050508188 /loss/FBt1/detection_loss‚2.6006512003285542 1loss/FBt3/assignment_loss‚0.04665425865864828 1loss/SRqqt4/detection_loss‚2.3280761105673653 epoch‚7ì¥ ±Ð‘Ò&‚(:† ¾êçÔÞؼ‰ gpu.0.gpu‚9.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚144.489' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.12225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›@_ |Ó&‚(:ñ ¾êçÔð× disk.sda1.in‚0 disk.sdb2.in‚5.40625 disk.sda1.out‚0 disk.rbd0.out‚ 3.087890625! disk.dm-0.out‚130497.24609375$ disk./.usageGB‚2975.089771270752 disk.sdb.in‚5.40625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda.out‚ 1180.8671875 disk.rbd2.in‚0# disk.nvme0n1.in‚329974.49609375 disk.dm-0.in‚531386.42578125 disk.sda.in‚ 5.56640625 disk.md0p1.out‚ 1137.99609375 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 3.025390625 cpu‚25.19317907225607) disk./.usagePercent‚83.18475517653428 disk.md0p1.in‚ 10.97265625 disk.sdb2.out‚ 1180.8671875 disk.md0.out‚ 1137.99609375" disk.nvme1n1.out‚ 66895.9765625 proc.cpu.threads‚17 disk.sda2.in‚ 5.56640625 disk.md0.in‚ 10.97265625# disk.nvme0n1.out‚63601.26953125 network.recv‚ 3.657332e+06) proc.memory.availableMB‚ 9948.08984375" proc.memory.rssMB‚ 3031.2421875) proc.memory.percent‚18.50123405456543 network.sent‚ 2.4040351e+07 disk.sda2.out‚ 1180.8671875" disk.nvme1n1.in‚201411.9296875% memory_percent‚39.281678199768066 disk.sdb.out‚ 1180.8671875‰ã%h’Ô&‚(:‡ ÍêçÔŠåÁ‡ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚140.145' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.03625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ýõ>†Õ&‚(:û ÍêçÔ†Óö‡) disk./.usagePercent‚83.18475677644226 disk.md0.in‚ 10.97265625 disk.md0.out‚ 1139.8359375! disk.sda.out‚1182.76123046875 disk.dm-0.out‚ 130617.421875 disk.sdb1.out‚0 disk.sda.in‚ 5.56640625# disk.nvme0n1.in‚329974.49609375 disk.sdb2.in‚5.40625 disk.rbd2.out‚ 3.025390625 disk.dm-0.in‚531386.42578125$ memory_percent‚39.28048610687256 proc.cpu.threads‚17 network.sent‚ 2.4126923e+07 network.recv‚ 3.664345e+06 disk.sda2.in‚ 5.56640625" disk.nvme1n1.out‚ 66944.0859375! disk.sdb.out‚1182.76123046875) proc.memory.availableMB‚ 9948.28515625$ disk./.usageGB‚2975.089828491211 disk.sda1.out‚0 cpu‚25.19295803924149" disk.nvme0n1.out‚ 63673.3359375" disk.sda2.out‚1182.76123046875 disk.md0p1.in‚ 10.97265625 disk.rbd0.out‚ 3.087890625 disk.rbd2.in‚0" disk.nvme1n1.in‚201411.9296875 disk.sdb1.in‚0" proc.memory.rssMB‚ 3031.2421875 disk.sda1.in‚0 disk.md0p1.out‚ 1139.8359375" disk.sdb2.out‚1182.76123046875 disk.rbd0.in‚0.015625 disk.sdb.in‚5.40625) proc.memory.percent‚18.50123405456543ë¼.i’Ö&‚(:‡ ÜêçÔËþª‡ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚134.575' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.64375 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÃAàÊŒ×&‚(:  ÜêçÔ¯Áá‡" disk.nvme0n1.out‚ 63745.6171875 disk.sdb2.in‚5.40625 disk.md0.out‚ 1141.29296875) proc.memory.percent‚18.50123405456543 disk.md0p1.in‚ 10.97265625 cpu‚25.192644651838396" disk.sdb2.out‚1184.26904296875 disk.md0.in‚ 10.97265625$ disk./.usageGB‚2975.089771270752 disk.rbd2.in‚0 disk.rbd2.out‚ 3.025390625" disk.nvme1n1.in‚201411.9296875! disk.sda.out‚1184.26904296875 disk.dm-0.in‚531386.42578125 disk.sda1.out‚0" disk.sda2.out‚1184.26904296875 network.recv‚ 3.668385e+06 disk.sda1.in‚0 disk.sda2.in‚ 5.56640625 disk.md0p1.out‚ 1141.29296875$ memory_percent‚39.28048610687256) proc.memory.availableMB‚ 9948.28515625) disk./.usagePercent‚83.18475517653428! disk.dm-0.out‚130785.84765625" proc.memory.rssMB‚ 3031.2421875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda.in‚ 5.56640625 proc.cpu.threads‚17# disk.nvme1n1.out‚67040.23046875# disk.nvme0n1.in‚329974.49609375 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 3.087890625 network.sent‚ 2.4132884e+07 disk.sdb.in‚5.40625! disk.sdb.out‚1184.26904296875Ýð’Ø&‚(:‡ ëêçÔ˱ù… gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚116.583' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.14575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÛöŠrÙ&‚(:ç ëêçԨśŠ disk.rbd2.in‚0 network.recv‚ 3.679e+06 disk.sda2.out‚ 1185.6796875" disk.nvme1n1.in‚201411.9296875) proc.memory.percent‚18.50123405456543 disk.sdb1.in‚0 disk.sdb2.out‚ 1185.6796875 disk.rbd2.out‚ 3.025390625$ memory_percent‚39.28050994873047& proc.memory.availableMB‚ 9948.28125" proc.memory.rssMB‚ 3031.2421875 disk.sda.in‚ 5.56640625# disk.nvme0n1.out‚63817.66796875 disk.md0p1.out‚ 1142.640625 disk.sdb1.out‚0 disk.dm-0.in‚531386.42578125% disk./.usageGB‚2974.8128089904785# disk.nvme0n1.in‚329974.49609375 disk.sda1.out‚0 disk.sda2.in‚ 5.56640625 disk.sdb2.in‚5.40625 disk.md0.out‚ 1142.640625 disk.sdb.in‚5.40625 proc.cpu.threads‚17 disk.sda1.in‚0 disk.md0p1.in‚ 10.97265625 disk.sdb.out‚ 1185.6796875 cpu‚25.192319390491097 network.sent‚ 2.4142289e+07 disk.dm-0.out‚130966.5546875 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 3.087890625# disk.nvme1n1.out‚67148.88671875( disk./.usagePercent‚83.1770111952601 disk.sda.out‚ 1185.6796875 disk.md0.in‚ 10.97265625¬F±É4Ú&/ _wandbruntime‚10785  _runtime‚10785ù<öÈÛ& ' lr-AdamW/pg2‚0.00013989572399917963 trainer/global_step‚9149 # _timestamp‚1.78847473193447e+09 _runtime‚10785.119545725 _step‚749 ' lr-AdamW/pg1‚0.00013989572399917963^ã ÍÜ&Ç _step‚749 ' lr-AdamW/pg1‚0.00013989572399917963 ' lr-AdamW/pg2‚0.00013989572399917963 trainer/global_step‚9149 # _timestamp‚1.78847473193447e+09 _runtime‚10785.119545725í––šÒ4Ý&/ _wandbruntime‚10785  _runtime‚10785­b”¯ñ'Þ&ëO @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚354688181898512 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚372743952625664 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18181493178368 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚413303543855616 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 174012203 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 174012201 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 248775019 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66052249 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 170051727 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚696 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 174012201 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 248801363 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 103998782 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚372818119082740 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18488879931392 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 trainer/global_step‚9149 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294813696 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚372743934622720 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 170051029 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18129954223588 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18488585117696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _runtime‚10785.121753425 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 248775019 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚390925679472640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚372743934622720 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 422813566 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18130183204672 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _step‚750 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚390925427801088 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 103998780 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚394814982678016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18181726846976 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318753792 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 174012203 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66052945 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 % _timestamp‚1.7884747319362354e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚372743952625664 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18181726846976 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0qµ|±ö'ß&ðO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 422813566 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18130183204672 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _step‚750 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚390925427801088 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 103998780 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚394814982678016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18181726846976 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318753792 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 174012203 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66052945 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 % _timestamp‚1.7884747319362354e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚372743952625664 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18181726846976 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚354688181898512 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚372743952625664 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18181493178368 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚413303543855616 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 174012203 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 174012201 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 248775019 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66052249 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 170051727 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚696 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 174012201 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 248801363 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 103998782 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚372818119082740 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18488879931392 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 trainer/global_step‚9149 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294813696 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚372743934622720 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 170051029 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18129954223588 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18488585117696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _runtime‚10785.121753425 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 248775019 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚390925679472640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚372743934622720îBõô½4à&/ _wandbruntime‚10786  _runtime‚1078695¤šá&ôM FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peaºi.`$k‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 248828081 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 170069776 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 170069086 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚394856642707968 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 248828081 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 174030478 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18183457204224 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚66060192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 104009584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 248801711 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚390966929548800 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 422832187 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 422858559 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _step‚751 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 248801711 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚688 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚372783238662656 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚690 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18131912712574 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18183690886144 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18490873542656 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚372857853949802 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 422858559 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18132141693866 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚10786.339578993 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 174030478 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 174030476 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚413347540190720 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚9149 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 $ _timestamp‚1.788474733153994e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 422832187 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18183690886144 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚413347223547392 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18490580839424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚390966677863936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚372783220659712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚372857607929150 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0[I^°ÿ&â&ùM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚372783238662656 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚690 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18131912712574 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18183690886144 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18490873542656 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚372857853949802 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 422858559 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18132141693866 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚10786.339578993 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 174030478 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 174030476 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚413347540190720 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚9149 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 $ _timestamp‚1.788474733153994e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 422832187 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18183690886144 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚413347223547392 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18490580839424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚390966677863936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚372783220659712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚372857607929150 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 248828081 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 170069776 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 170069086 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚394856642707968 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 248828081 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 174030478 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18183457204224 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚66060192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 104009584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 248801711 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚390966929548800 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 422832187 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 422858559 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _step‚751 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 248801711 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚688 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ï¶çÆ84ã&/ _wandbruntime‚10786  _runtime‚10786tNeû`ä&Ú 0loss/FBt4/assignment_loss‚0.2648299061926075 /loss/FRt2/detection_loss‚2.1110524946061724 1loss/SRqqt4/detection_loss‚2.1904963574239185 /loss/FBt4/detection_loss‚2.8186439633369447 2loss/SRqqt1/assignment_loss‚1.3766473482939459 /loss/FRt4/detection_loss‚3.2948942496017977 1loss/SRqqt3/detection_loss‚0.9008942405644216 /loss/FBt3/detection_loss‚0.5549320604938727 trainer/global_step‚9149 _runtime‚10786.342359925 0loss/FRt4/assignment_loss‚2.8163754587823693 2loss/SRqqt4/assignment_loss‚1.2837781086564064 1loss/FBt2/assignment_loss‚0.06190512305801504 1loss/SRqqt2/detection_loss‚2.2589744701981544 -loss/FBt1/detection_loss‚3.02181813120842 _step‚752 1loss/SRqqt3/assignment_loss‚1.000682590343058 2loss/FBt3/assignment_loss‚0.012143896787257255 /loss/FRt1/detection_loss‚2.6155072579512724 1loss/SRqqt1/detection_loss‚1.8161303571292333 1loss/FBt1/assignment_loss‚0.03457681817219793 epoch‚7 % _timestamp‚1.7884747331573873e+09 0loss/FRt3/assignment_loss‚1.4530335412898534 1loss/SRqqt2/assignment_loss‚1.574063833969246 &loss/total_loss‚40.994502763778975 0loss/FRt2/assignment_loss‚3.1801361690868033 /loss/FRt3/detection_loss‚1.2759565108416366 /loss/FBt2/detection_loss‚2.0237838964049635 /loss/FRt1/assignment_loss‚3.053245979386407¬.Z8eå&ß -loss/FBt1/detection_loss‚3.02181813120842 _step‚752 1loss/SRqqt3/assignment_loss‚1.000682590343058 2loss/FBt3/assignment_loss‚0.012143896787257255 /loss/FRt1/detection_loss‚2.6155072579512724 1loss/SRqqt1/detection_loss‚1.8161303571292333 1loss/FBt1/assignment_loss‚0.03457681817219793 epoch‚7 % _timestamp‚1.7884747331573873e+09 0loss/FRt3/assignment_loss‚1.4530335412898534 1loss/SRqqt2/assignment_loss‚1.574063833969246 &loss/total_loss‚40.994502763778975 0loss/FRt2/assignment_loss‚3.1801361690868033 /loss/FRt3/detection_loss‚1.2759565108416366 /loss/FBt2/detection_loss‚2.0237838964049635 /loss/FRt1/assignment_loss‚3.053245979386407 0loss/FBt4/assignment_loss‚0.2648299061926075 /loss/FRt2/detection_loss‚2.1110524946061724 1loss/SRqqt4/detection_loss‚2.1904963574239185 /loss/FBt4/detection_loss‚2.8186439633369447 2loss/SRqqt1/assignment_loss‚1.3766473482939459 /loss/FRt4/detection_loss‚3.2948942496017977 1loss/SRqqt3/detection_loss‚0.9008942405644216 /loss/FBt3/detection_loss‚0.5549320604938727 trainer/global_step‚9149 _runtime‚10786.342359925 0loss/FRt4/assignment_loss‚2.8163754587823693 2loss/SRqqt4/assignment_loss‚1.2837781086564064 1loss/FBt2/assignment_loss‚0.06190512305801504 1loss/SRqqt2/detection_loss‚2.2589744701981544ð-'Næ&‚(:„ úêçÔÿ™×… gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚90.236' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚22.559 gpu.0.smClock‚1200 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U¸uƒç&‚(:ø úêçÔÉ‹‹ network.sent‚ 2.415925e+07$ disk./.usageGB‚2974.772773742676! disk.sdb2.out‚1187.6435546875 disk.sda.in‚ 5.56640625$ memory_percent‚39.28074836730957 proc.cpu.threads‚17 network.recv‚ 3.683167e+06 disk.sdb2.in‚5.40625 disk.rbd0.in‚0.015625 disk.rbd2.in‚0# disk.nvme1n1.out‚67188.22265625! disk.sda2.out‚1187.6435546875 disk.md0p1.in‚ 10.97265625 disk.rbd2.out‚ 3.025390625 disk.sdb1.out‚0# disk.nvme0n1.in‚329974.49609375 disk.md0.out‚ 1144.5546875" disk.nvme1n1.in‚201411.9296875" proc.memory.rssMB‚ 3031.2421875 disk.sda1.out‚0 disk.sda2.in‚ 5.56640625 disk.md0.in‚ 10.97265625 disk.rbd0.out‚ 3.087890625 disk.sdb.in‚5.40625 disk.sdb1.in‚0 disk.sda.out‚1187.6435546875# disk.nvme0n1.out‚63889.81640625) disk./.usagePercent‚83.17589179297549 disk.dm-0.in‚531386.42578125 disk.dm-0.out‚131078.0390625) proc.memory.percent‚18.50123405456543 disk.md0p1.out‚ 1144.5546875 disk.sdb.out‚1187.6435546875 disk.sda1.in‚0( proc.memory.availableMB‚ 9948.2421875 cpu‚25.19197979104589C~ØÍè&‚(:… ‰ëçÔÛÉ⇠gpu.0.gpu‚27.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚132.732' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.183 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0húŠ0„é&‚(:ù ‰ëçÔŸ…‰# disk.nvme0n1.in‚329974.49609375! disk.nvme0n1.out‚ 63961.859375 disk.md0p1.out‚ 1146.02734375$ memory_percent‚39.28227424621582 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 3.087890625" disk.nvme1n1.in‚201411.9296875 disk.nvme1n1.out‚ 67255.125! disk.sdb.out‚1189.17626953125 disk.sdb1.in‚0" proc.memory.rssMB‚ 3031.2421875 network.sent‚ 2.4254123e+07 disk.sdb2.in‚5.40625 disk.md0.out‚ 1146.02734375 disk.sda.in‚ 5.56640625 disk.sda1.out‚0 cpu‚25.191700697978465 network.recv‚ 3.690348e+06 disk.dm-0.out‚ 131216.984375 disk.sda2.in‚ 5.56640625) disk./.usagePercent‚83.17776965830403" disk.sdb2.out‚1189.17626953125 disk.sda1.in‚0( proc.memory.availableMB‚ 9947.9921875 proc.cpu.threads‚17% disk./.usageGB‚2974.8399353027344 disk.dm-0.in‚531386.42578125 disk.md0.in‚ 10.97265625 disk.sdb.in‚5.40625" disk.sda2.out‚1189.17626953125 disk.sdb1.out‚0! disk.sda.out‚1189.17626953125 disk.md0p1.in‚ 10.97265625) proc.memory.percent‚18.50123405456543 disk.rbd2.in‚0 disk.rbd2.out‚ 3.0253906255@Kô’ê&‚(:‡ ˜ëçԯ gpu.0.gpu‚11.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚142.313' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.57825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í+·|ë&‚(:ñ ˜ëçÔÖ—Õ‡" proc.memory.rssMB‚ 3031.2421875 disk.sdb.out‚1190.794921875 disk.sdb1.in‚0 disk.sda.out‚1190.794921875 disk.dm-0.out‚131360.0859375 disk.md0p1.in‚ 10.97265625 disk.sdb2.out‚1190.794921875 network.sent‚ 2.4259622e+07 disk.rbd0.out‚ 3.087890625 disk.sdb1.out‚0 disk.sda.in‚ 5.56640625 network.recv‚ 3.694528e+06 disk.md0.in‚ 10.97265625 disk.md0.out‚ 1147.5703125 disk.sda1.in‚0) proc.memory.percent‚18.50123405456543 disk.rbd2.in‚0 proc.cpu.threads‚17# disk.nvme0n1.in‚329974.49609375 disk.sda1.out‚0 disk.md0p1.out‚ 1147.5703125 disk.rbd2.out‚ 3.025390625" disk.nvme1n1.in‚201411.9296875! disk.nvme1n1.out‚ 67326.109375" disk.nvme0n1.out‚ 64033.9765625 disk.dm-0.in‚531386.42578125) disk./.usagePercent‚83.18123719220331# disk./.usageGB‚2974.96395111084 disk.rbd0.in‚0.015625 disk.sdb.in‚5.40625 disk.sda2.in‚ 5.56640625 cpu‚25.19146610570811 disk.sda2.out‚1190.794921875 disk.sdb2.in‚5.40625$ memory_percent‚39.28370475769043( proc.memory.availableMB‚ 9947.7578125¹²nú4ì&/ _wandbruntime‚10844  _runtime‚10844ÇšWÊí&Ä ' lr-AdamW/pg1‚0.00013545682240539237 ' lr-AdamW/pg2‚0.00013545682240539237 trainer/global_step‚9199 % _timestamp‚1.7884747910400045e+09 _runtime‚10844.225037485 _step‚753Þ÷j(Ïî&É ' lr-AdamW/pg1‚0.00013545682240539237 ' lr-AdamW/pg2‚0.00013545682240539237 trainer/global_step‚9199 % _timestamp‚1.7884747910400045e+09 _runtime‚10844.225037485 _step‚753ñM \4ï&/ _wandbruntime‚10844  _runtime‚10844Ñ Ôð&ëO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚392987945357312 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 _step‚754 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚392988197028864 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚374792574994448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18588648905728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚374708254475264 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325045248 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66403284 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚392987945357312 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18279942553600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚9199 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚374792821014892 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 250¼ñ‹šî112971 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚396899157240320 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 174925951 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚695 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚693 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚356564682730352 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18228121245180 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 250139315 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚356564699769712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 425038922 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18279708884992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 174925953 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚374708254475264 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18588347800576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18227892264096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚374708236472320 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚396899181180416 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582ç2vËö'ñ&ðO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 174925953 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚374708254475264 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18588347800576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18227892264096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚374708236472320 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚396899181180416 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚392987945357312 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 _step‚754 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚392988197028864 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚374792574994448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18588648905728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚374708254475264 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325045248 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66403284 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚392987945357312 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18279942553600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚9199 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚374792821014892 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 250112971 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚396899157240320 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 174925951 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚695 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚693 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚356564682730352 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18228121245180 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 250139315 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚356564699769712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 425038922 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18279708884992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136ò4ykó’ò&‚(:‡ §ëçÔр߇ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚129.005' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.25125 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§wIÏŠó&‚(:ÿ §ëçÔ îˆ disk.sdb1.in‚0# disk.nvme0n1.in‚329974.49609375 disk.dm-0.out‚ 131524.015625 disk.sda2.in‚ 5.56640625 disk.md0p1.in‚ 10.97265625 disk.sdb2.in‚5.40625 disk.rbd0.out‚ 3.087890625# disk.nvme0n1.out‚64106.02734375) disk./.usagePercent‚83.18478674805179% disk./.usageGB‚2975.0909004211426" disk.nvme1n1.in‚201411.9296875 disk.sdb1.out‚0 disk.dm-0.in‚531386.42578125# disk.nvme1n1.out‚67417.98828125! disk.sdb.out‚1192.70263671875" disk.sdb2.out‚1192.70263671875 network.recv‚ 3.69833e+06! disk.sda.out‚1192.70263671875 disk.md0.in‚ 10.97265625 disk.rbd2.in‚0 disk.rbd2.out‚ 3.025390625" proc.memory.rssMB‚ 3031.2421875 disk.sda1.out‚0 disk.rbd0.in‚0.015625) proc.memory.percent‚18.50123405456543 proc.cpu.threads‚17 network.sent‚ 2.4264877e+07 disk.sda.in‚ 5.56640625 disk.sda1.in‚0" disk.sda2.out‚1192.70263671875 disk.md0p1.out‚ 1149.41796875 disk.sdb.in‚5.40625 disk.md0.out‚ 1149.41796875) proc.memory.availableMB‚ 9947.76171875$ memory_percent‚39.28368091583252 cpu‚25.19122090307749—ƒà4ô&/ _wandbruntime‚10845  _runtime‚10845úXÀû&õ&õM ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18281906592768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 250139663 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 425083889 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 250166033 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 174944228 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚396940865150464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚356602230127136 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 250166033 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚66410690 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18590343344128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚415531499100672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚356602213087776 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 174944228 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18229850753082 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 425110261 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚374832309861510 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚393029447105024 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 425110261 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚687 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 170959685 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚66410003 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚396940841210368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18230079734374 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 _step‚755 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 174944226 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 170960374 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18590633950208 % _timestamp‚1.7884747921801965e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 250139663 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚393029195420160 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 104549684 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚9199 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚66410690 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18590343344128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚415531499100672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚356602213087776 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 174944228 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18229850753082 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 425110261 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚374832309861510 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚393029447105024 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 425110261 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚687 D>DeviceStatsMonitor.on_train_batch_end/segmæ^'É`ent.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 170959685 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚66410003 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚396940841210368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18230079734374 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 _step‚755 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 174944226 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 170960374 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18590633950208 % _timestamp‚1.7884747921801965e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 250139663 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚393029195420160 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 104549684 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚9199 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚758 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚395050462913536 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚376672556324864 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 251477267 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 175839701 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18377924591616 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18378158260224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚417671440126976 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18688372906496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚9249 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚376672538321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18377924591616 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚417671752589312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 427316970 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 $ _timestamp‚1.788474846144344e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 105078982 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18326059285688 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 171836234 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚395050714585088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚376672556324864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚398983379682816 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚358441217640912 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚758 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚395050462913536 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚376672556324864 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 251477267 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 175839701 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18377924591616 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18378158260224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚417671440126976 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18688372906496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚9249 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚376672538321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18377924591616 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚417671752589312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 427316970 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 $ _timestamp‚1.788474846144344e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 105078982 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18326059285688 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 171836234 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚395050714585088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚376672556324864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚398983379682816 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚358441217640912 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚376711842361856 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.p·jÄ9&eak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚376711824358912 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 427361963 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 105089784 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18327788793590 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚376711842361856 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18690074732032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 251477615 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚358478747998336 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚395091712976384 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 105089782 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚417715114444800 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚376711824358912 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 251477615 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 427335591 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 175857978 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚399025039712768 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 427361963 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18380122299392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 251503985 trainer/global_step‚9249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚395091964661248 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 171853834 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚399025063652864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18379888617472 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚376806765773218 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 251503985 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 175857976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 % _timestamp‚1.7884748473480604e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18690367435264 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 175857976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚10900.534227938µ:m'ˆ'úM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 427361963 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18380122299392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 251503985 trainer/global_step‚9249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚395091964661248 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 171853834 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚399025063652864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18379888617472 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚376806765773218 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 251503985 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 175857976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 % _timestamp‚1.7884748473480604e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18690367435264 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 175857976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚10900.534227938 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚376711842361856 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚376711824358912 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 427361963 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 105089784 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18327788793590 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚376711842361856 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18690074732032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 251477615 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚358478747998336 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚395091712976384 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 105089782 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚417715114444800 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚376711824358912 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 251477615 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 427335591 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 175857978 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚399025039712768÷Jˆ¨á4‰'/ _wandbruntime‚10900  _runtime‚10900?.UŠ'Ï 1loss/FBt1/assignment_loss‚0.45693031712994525 0loss/FBt4/assignment_loss‚0.1310833585269446 /loss/FBt2/detection_loss‚2.1204324109213695 _step‚760 .loss/FRt2/detection_loss‚2.095927662346653 /loss/SRqqt4/detection_loss‚2.85274216708015 _runtime‚10900.535813545 2loss/SRqqt2/assignment_loss‚1.3777126469782421 .loss/FBt1/detection_loss‚2.441291019320488 /loss/FRt1/assignment_loss‚2.781751755211088 0loss/FBt2/assignment_loss‚0.1416448322814229 .loss/FRt1/detection_loss‚2.650960971083906 /loss/FRt3/detection_loss‚1.7124323581135081 2loss/SRqqt1/assignment_loss‚2.0885118690172306 0loss/FBt3/assignment_loss‚0.0305671106377605 1loss/SRqqt3/detection_loss‚0.8523741485647012 $ _timestamp‚1.788474847350813e+09 trainer/global_step‚9249 0loss/FRt2/assignment_loss‚3.5099959427660163 0loss/FRt3/assignment_loss‚1.1475623807599467 0loss/SRqqt4/assignment_loss‚1.91249844935887 /loss/FRt4/detection_loss‚3.3425415257612863 1loss/SRqqt1/detection_loss‚2.1107828483769766 0loss/SRqqt2/detection_loss‚2.071004772186279 %loss/total_loss‚42.65838286915884 .loss/FRt4/assignment_loss‚2.99281773964564 2loss/SRqqt3/assignment_loss‚0.9278958423360189 /loss/FBt3/detection_loss‚0.5457947516840869 /loss/FBt4/detection_loss‚2.3631259890703054 epoch‚7ø¥)Z‹'Ô /loss/FRt4/detection_loss‚3.3425415257612863 1loss/SRqqt1/detection_loss‚2.1107828483769766 0loss/SRqqt2/detection_loss‚2.071004772186279 %loss/total_loss‚42.65838286915884 .loss/FRt4/assignment_loss‚2.99281773964564 2loss/SRqqt3/assignment_loss‚0.9278958423360189 /loss/FBt3/detection_loss‚0.5457947516840869 /loss/FBt4/detection_loss‚2.3631259890703054 epoch‚7 1loss/FBt1/assignment_loss‚0.45693031712994525 0loss/FBt4/assignment_loss‚0.1310833585269446 /loss/FBt2/detection_loss‚2.1204324109213695 _step‚760 .loss/FRt2/detection_loss‚2.095927662346653 /loss/SRqqt4/detection_loss‚2.85274216708015 _runtime‚10900.535813545 2loss/SRqqt2/assignment_loss‚1.3777126469782421 .loss/FBt1/detection_loss‚2.441291019320488 /loss/FRt1/assignment_loss‚2.781751755211088 0loss/FBt2/assignment_loss‚0.1416448322814229 .loss/FRt1/detection_loss‚2.650960971083906 /loss/FRt3/detection_loss‚1.7124323581135081 2loss/SRqqt1/assignment_loss‚2.0885118690172306 0loss/FBt3/assignment_loss‚0.0305671106377605 1loss/SRqqt3/detection_loss‚0.8523741485647012 $ _timestamp‚1.788474847350813e+09 trainer/global_step‚9249 0loss/FRt2/assignment_loss‚3.5099959427660163 0loss/FRt3/assignment_loss‚1.1475623807599467 0loss/SRqqt4/assignment_loss‚1.91249844935887ø Ï’Œ'‚(:‡ ãëçÔ’õˆ gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚111.955' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚27.98875 gpu.0.smClock‚1260 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ß=ÜYˆ'‚(:ý ãëçÔúǵˆ disk.dm-0.out‚132177.1953125 disk.sda1.in‚0 disk.md0p1.out‚ 1155.9453125 network.sent‚ 2.4486228e+07 disk.sdb.in‚5.40625 proc.cpu.threads‚17 disk.sdb1.out‚0% disk./.usageGB‚2974.8395347595215 disk.sdb2.in‚5.40625 disk.md0.in‚ 10.97265625 disk.rbd0.out‚ 3.087890625 disk.rbd2.in‚0# disk.nvme0n1.in‚329974.49609375) disk./.usagePercent‚83.17775845894815! disk.sdb2.out‚1199.4697265625 disk.rbd0.in‚0.015625 disk.sda.out‚1199.4697265625) proc.memory.percent‚18.50123405456543 cpu‚25.190263888199635# disk.nvme0n1.out‚64466.60546875% memory_percent‚39.284634590148926) proc.memory.availableMB‚ 9947.60546875 network.recv‚ 3.727062e+06 disk.sdb1.in‚0" proc.memory.rssMB‚ 3031.2421875" disk.nvme1n1.in‚201411.9296875# disk.nvme1n1.out‚67710.58984375 disk.sdb.out‚1199.4697265625 disk.sda1.out‚0 disk.md0p1.in‚ 10.97265625 disk.rbd2.out‚ 3.025390625 disk.dm-0.in‚531386.42578125 disk.sda2.in‚ 5.56640625! disk.sda2.out‚1199.4697265625 disk.md0.out‚ 1155.9453125 disk.sda.in‚ 5.56640625õù±‘Ž'‚(:† òëçÔÒËœŠ gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚129.282' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.3205 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%Øò}'‚(:ò òëçÔžø›$ memory_percent‚39.28477764129639) proc.memory.availableMB‚ 9947.58203125) proc.memory.percent‚18.50123405456543 disk.sdb.in‚5.40625# disk.nvme0n1.in‚329974.49609375 network.recv‚ 3.734428e+06 disk.sda2.in‚ 5.56640625 disk.md0p1.in‚ 10.97265625 disk.sdb2.out‚ 1201.0546875 disk.md0.in‚ 10.97265625 disk.md0.out‚ 1157.47265625 disk.sdb1.out‚0 disk.dm-0.in‚531386.42578125 disk.rbd0.out‚ 3.087890625 disk.sdb1.in‚0" proc.memory.rssMB‚ 3031.2421875) disk./.usagePercent‚83.17867829937717# disk.nvme0n1.out‚64538.73046875 disk.rbd2.in‚0 disk.rbd2.out‚ 3.025390625 disk.sdb.out‚ 1201.0546875 network.sent‚ 2.4584316e+07 disk.sda.in‚ 5.56640625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0.015625" disk.nvme1n1.in‚201411.9296875" disk.nvme1n1.out‚ 67777.8828125 cpu‚25.189988772965407% disk./.usageGB‚2974.8724327087402 disk.sda.out‚ 1201.0546875! disk.dm-0.out‚132316.61328125 disk.sda2.out‚ 1201.0546875 disk.md0p1.out‚ 1157.47265625 disk.sdb2.in‚5.40625 proc.cpu.threads‚17ï³dI'‚(:… ìçÔߪç… gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚139.868' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.967 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 »(õ…‘'‚(:ú ìçÔË’ˆ‡ disk.sda1.out‚0 disk.sdb2.in‚5.40625 network.sent‚ 2.4587213e+07 disk.sdb1.out‚0! disk.sda.out‚1202.64111328125( proc.memory.availableMB‚ 9947.5703125" proc.memory.rssMB‚ 3031.2421875" disk.nvme1n1.in‚201411.9296875 disk.sdb1.in‚0 disk.sda2.in‚ 5.56640625" disk.sda2.out‚1202.64111328125 disk.rbd0.in‚0.015625 network.recv‚ 3.738719e+06 disk.sdb.in‚5.40625 disk.md0.out‚ 1158.98828125 disk.rbd0.out‚ 3.087890625) disk./.usagePercent‚83.18480552030543 disk.nvme1n1.out‚67848.75 disk.dm-0.in‚531386.42578125 disk.md0p1.out‚ 1158.98828125 disk.rbd2.in‚0" disk.sdb2.out‚1202.64111328125" disk.nvme0n1.out‚ 64610.8046875% disk./.usageGB‚2975.0915718078613! disk.sdb.out‚1202.64111328125 disk.sda.in‚ 5.56640625 disk.sda1.in‚0$ memory_percent‚39.28484916687012 cpu‚25.189704431546716# disk.nvme0n1.in‚329974.49609375 disk.dm-0.out‚132459.5546875 disk.rbd2.out‚ 3.025390625) proc.memory.percent‚18.50123405456543 proc.cpu.threads‚17 disk.md0p1.in‚ 10.97265625 disk.md0.in‚ 10.97265625/ï|’’'‚(:‡ ìçԥϢ‡ gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚139.251' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.81275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e½ìç„“'‚(:ù ìçÔøǶˆ" disk.nvme1n1.in‚201411.9296875 disk.sda1.in‚0" proc.memory.rssMB‚ 3031.2421875! disk.sdb2.out‚1204.7529296875 disk.sda2.in‚ 5.56640625 disk.md0p1.in‚ 10.97265625 disk.sdb2.in‚5.40625 disk.md0.in‚ 10.97265625 disk.rbd0.out‚ 3.087890625 disk.rbd2.in‚0 disk.sda.out‚1204.7529296875" disk.nvme0n1.out‚ 64682.9453125# memory_percent‚39.2848014831543' proc.memory.availableMB‚ 9947.578125) disk./.usagePercent‚83.18478952122562% disk./.usageGB‚2975.0909996032715 disk.rbd2.out‚ 3.025390625# disk.nvme0n1.in‚329974.49609375) proc.memory.percent‚18.50123405456543 disk.rbd0.in‚0.015625 disk.sdb.in‚5.40625 disk.dm-0.in‚531386.42578125! disk.dm-0.out‚132675.72265625! disk.sda2.out‚1204.7529296875 cpu‚25.18946158972916 proc.cpu.threads‚17 network.recv‚ 3.74389e+06 disk.sdb1.out‚0 disk.sda1.out‚0 disk.md0p1.out‚ 1161.02734375 network.sent‚ 2.4594057e+07# disk.nvme1n1.out‚67992.77734375 disk.sdb.out‚1204.7529296875 disk.sda.in‚ 5.56640625 disk.md0.out‚ 1161.02734375 disk.sdb1.in‚07Æ’c4”'/ _wandbruntime‚10960  _runtime‚109601!q)É•'à _runtime‚10960.220591113 _step‚761 ' lr-AdamW/pg1‚0.00012675296732300034 ' lr-AdamW/pg2‚0.00012675296732300034 trainer/global_step‚9299 $ _timestamp‚1.788474907035527e+09‘²vWΖ'È ' lr-AdamW/pg1‚0.00012675296732300034 ' lr-AdamW/pg2‚0.00012675296732300034 trainer/global_step‚9299 $ _timestamp‚1.788474907035527e+09 _runtime‚10960.220591113 _step‚761ùÃÔ æ4—'/ _wandbruntime‚10960  _runtime‚10960Üf”&F ˜'ëO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚397112980469760 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18476140298240 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67108931 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 252815219 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 176753453 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67108244 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 105619080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚378741486817864 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 429568672 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚401067578185216 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 105619082 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚378636858174464 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚378636840171520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚397112980469760 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18476373966848 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 176753453 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 252788875 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 172727324 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚419855495000064 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚419855809559552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚378636840171520 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚397113232141312 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 429542326 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18476373966848 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0Kô¬ö'™'ðO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 429542326 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18476373966848 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚397112980469760 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18476140298240 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67108931 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 252815219 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 176753453 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67108244 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 105619080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚378741486817864 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 429568672 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚401067578185216 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 105619082 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚378636858174464 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚378636840171520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚397112980469760 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18476373966848 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 176753453 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 252788875 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 172727324 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚419855495000064 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚419855809559552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚378636840171520 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚397113232141312 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136úKÕ½4š'/ _wandbruntime‚10961  _runtime‚10961]uRœú&›'ôM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 252841937 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 429613665 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 176771728 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 429587293 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 252815567 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18425726834098 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 252815567 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 105629884 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚378676126208512 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚397154482217472 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 429587293 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67115427 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 176771728 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚378781221684926 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18478104324096 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 105629882 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _step‚763 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚378676144211456 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚378676126208512 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚681 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18425955815390 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18478104324096 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 176771726 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚401109238215168 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18790224812544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚683 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18789934206464 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 $ _timestamp‚1.788474908278646e+09 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 172745309 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚360355265869536 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚9299 _runtime‚10961.465401661 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚419899486967808 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 172745309 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚360355265869536 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚9299 _runtime‚10961.465401661 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚419899486967808 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 252841937 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 429613665 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 176771728 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 429587293 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 252815567 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.s1Õ[ñmall_pool.peak‚ 565025792 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18425726834098 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 252815567 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 105629884 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚378676126208512 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚397154482217472 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 429587293 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67115427 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 176771728 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚378781221684926 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18478104324096 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 105629882 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _step‚763 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚378676144211456 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚378676126208512 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚681 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18425955815390 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18478104324096 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 176771726 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚401109238215168 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18790224812544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚683 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18789934206464 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 $ _timestamp‚1.788474908278646e+09 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 177667201 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚380601160024064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 173617455 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 173618155 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚403151752747520 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18521935366704 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 254126827 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 431820374 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _runtime‚11015.279905556 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚422039556666368 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18574356004864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18574356004864 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18574589673472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚380715942729572 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚362194253383312 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 431794028 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 177667201 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 254153171 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 % _timestamp‚1.7884749620946493e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚399175498025984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚399175498025984 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚380601142021120 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚380716188750016 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚380601160024064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 106159182 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚403151776687616 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 254126827 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67458973 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚698 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚399175749697536 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚380601142021120 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18888086149632 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18574589673472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67458275 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 trainer/global_step‚9349 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 177667203 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚700-çƒ!­'ðO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚362194236343952 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 254153171 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚399175749697536 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282230784 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _step‚766 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚380601160024064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 173617455 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 173618155 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚403151752747520 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18521935366704 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 254126827 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 431820374 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _runtime‚11015.279905556 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚422039556666368 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18574356004864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18574356004864 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18574589673472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚380715942729572 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚362194253383312 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 431794028 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 177667201 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 254153171 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 % _timestamp‚1.7884749620946493e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚399175498025984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚399175498025984 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚380601142021120 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚380716188750016 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚380601160024064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 106159182 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚403151776687616 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 254126827 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67458973 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚698 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚399175749697536 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚380601142021120 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18888086149632 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18574589673472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67458275 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 trainer/global_step‚9349 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 177667203 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚700 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 JDeviLXñnsceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 177667201 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 B„m4®'/ _wandbruntime‚11016  _runtime‚11016•\—sû&¯'õM L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 173635534 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 254153519 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚362231766701376 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 431865367 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 254153519 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚422083546088960 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _step‚767 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 431838995 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18523893855898 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 431865367 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚399216999773696 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚380640446061056 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 106169984 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚399216999773696 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 177685476 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚9349 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18576553712640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚380640428058112 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚403193460657664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚380640428058112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 177685476 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚380640446061056 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 431838995 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18576320030720.:­é'°'úM OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 177685478 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 177685478 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚380755431575982 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 _runtime‚11016.249398652 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚403193436717568 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18523664874606 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚422083237834240 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚399216748088832 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18576553712640 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 254179889 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 106169984 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚399216999773696 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 177685476 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚9349 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18576553712640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚380640428058112 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚403193460657664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚380640428058112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 177685476 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚380640446061056 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 431838995 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18576320030720 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 173635534 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 254153519 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚362231766701376 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 431865367 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 254153519 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚422083546088960 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _step‚767 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 431838995 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18523893855898 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 431865367 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚399216999773696 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚380640446061056 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚401238015582208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 255491123 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 174507676 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚401238015582208 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚424223597646336 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18619873407212 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18672805380096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18672571711488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚382565461873664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 178580953 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚382565443870720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67808396 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚364070771254512 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚382565461873664 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 106699280 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18619644426128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 434072076 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 174508363 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚364070754215152 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18672805380096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚687 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚424223908011520 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚9399C- ¢ö'Á'ðO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚424223597646336 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18619873407212 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18672805380096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18672571711488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚382565461873664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 178580953 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚382565443870720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67808396 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚364070771254512 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚382565461873664 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 106699280 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18619644426128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 434072076 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 174508363 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚364070754215152 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18672805380096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚687 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚424223908011520 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚9399 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚401238015582208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 255491123 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 174507676 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚401238015582208‚?•Ï»4Â'/ _wandbruntime‚11071  _runtime‚11071,¡¿éû&Ã'õM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚382604747910656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 174525913 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 434117069 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 178599226 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 255517841 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 434090697 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚382604747910656 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 106710084 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚401279517329920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚382730133508342 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18621831896406 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 255491471 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 174525247 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚424267584939520 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚424267278781952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 255491471 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚664 trainer/global_step‚9399 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚401279265645056 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 255517841 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 106710082 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 % _timestamp‚1.7884750184268577e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚405277659160064 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18989925779456 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67815165 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 434090697 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18674535737344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18621602915114 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18674769419264 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚771 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚401279517329920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 434117069 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67815829 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚382604729907712 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18674769419264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚11071.612498407 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚666 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚405277635219968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚5820å_LÄ'úM OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67815165 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 434090697 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18674535737344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18621602915114 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18674769419264 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚771 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚401279517329920 L>DeviceStatsMonitoÖj!´r.on_train_batch_end/allocation.all.allocated‚ 434117069 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67815829 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚382604729907712 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18674769419264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚11071.612498407 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚666 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚405277635219968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚382604747910656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 174525913 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 434117069 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 178599226 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 255517841 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 434090697 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚382604747910656 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 106710084 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚401279517329920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚382730133508342 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18621831896406 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 255491471 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 174525247 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚424267584939520 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚424267278781952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 255491471 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚664 trainer/global_step‚9399 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚401279265645056 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 255517841 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 106710082 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 % _timestamp‚1.7884750184268577e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚405277659160064 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18989925779456ƒEä 4Å'/ _wandbruntime‚11071  _runtime‚11071À² XÆ'Ò .loss/FBt2/detection_loss‚2.216731770680501 /loss/FRt4/assignment_loss‚3.105316611131032 2loss/SRqqt3/assignment_loss‚0.7750244582127227 /loss/FRt2/detection_loss‚1.9967484597430525 /loss/FBt1/detection_loss‚2.5711019310084255 trainer/global_step‚9399 /loss/FRt2/assignment_loss‚3.778412421112475 .loss/FRt3/detection_loss‚1.480631863258572 epoch‚7 _runtime‚11071.614111394 0loss/FRt1/assignment_loss‚2.2982603020784333 1loss/FBt4/assignment_loss‚0.19305933901523473 /loss/SRqqt2/detection_loss‚2.57537625481685 .loss/SRqqt4/detection_loss‚2.6219602227211 /loss/FBt3/detection_loss‚0.5987062428241082 &loss/total_loss‚40.833623117880734 2loss/SRqqt2/assignment_loss‚1.9464778742440103 1loss/FBt1/assignment_loss‚0.08792340131035106 1loss/SRqqt3/detection_loss‚0.7934886490981473 2loss/SRqqt1/assignment_loss‚1.1949533715489364 .loss/FRt1/detection_loss‚2.546605754189375 /loss/SRqqt1/detection_loss‚1.89886354974338 % _timestamp‚1.7884750184294136e+09 0loss/FRt3/assignment_loss‚1.1755915482838948 1loss/FBt3/assignment_loss‚0.09540804487507572 /loss/FBt4/detection_loss‚1.9477389628688495 2loss/SRqqt4/assignment_loss‚1.8128210739655928 1loss/FBt2/assignment_loss‚0.11603419266217357 .loss/FRt4/detection_loss‚3.006386818488439 _step‚772¨‘™]Ç'× /loss/FBt4/detection_loss‚1.9477389628688495 2loss/SRqqt4/assignment_loss‚1.8128210739655928 1loss/FBt2/assignment_loss‚0.11603419266217357 .loss/FRt4/detection_loss‚3.006386818488439 _step‚772 .loss/FBt2/detection_loss‚2.216731770680501 /loss/FRt4/assignment_loss‚3.105316611131032 2loss/SRqqt3/assignment_loss‚0.7750244582127227 /loss/FRt2/detection_loss‚1.9967484597430525 /loss/FBt1/detection_loss‚2.5711019310084255 trainer/global_step‚9399 /loss/FRt2/assignment_loss‚3.778412421112475 .loss/FRt3/detection_loss‚1.480631863258572 epoch‚7 _runtime‚11071.614111394 0loss/FRt1/assignment_loss‚2.2982603020784333 1loss/FBt4/assignment_loss‚0.19305933901523473 /loss/SRqqt2/detection_loss‚2.57537625481685 .loss/SRqqt4/detection_loss‚2.6219602227211 /loss/FBt3/detection_loss‚0.5987062428241082 &loss/total_loss‚40.833623117880734 2loss/SRqqt2/assignment_loss‚1.9464778742440103 1loss/FBt1/assignment_loss‚0.08792340131035106 1loss/SRqqt3/detection_loss‚0.7934886490981473 2loss/SRqqt1/assignment_loss‚1.1949533715489364 .loss/FRt1/detection_loss‚2.546605754189375 /loss/SRqqt1/detection_loss‚1.89886354974338 % _timestamp‚1.7884750184294136e+09 0loss/FRt3/assignment_loss‚1.1755915482838948 1loss/FBt3/assignment_loss‚0.09540804487507572„™cl£’È'‚(:‡ —íçÔ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚139.251' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.81275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£×â ‹É'‚(:€  —íçԀ览" disk.sda2.out‚1219.27880859375 disk.md0.out‚ 1174.99609375 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.rbd2.out‚ 3.025390625 disk.sdb1.in‚0 proc.cpu.threads‚17 disk.sdb2.in‚5.40625 disk.sdb1.out‚0! disk.sda.out‚1219.27880859375 network.recv‚ 3.801708e+06 disk.dm-0.in‚531405.14453125! disk.nvme1n1.out‚ 68235.203125! disk.sdb.out‚1219.27880859375 cpu‚25.187431465473686! disk.dm-0.out‚133728.44921875 disk.sda1.out‚0 disk.sda2.in‚ 5.56640625" disk.sdb2.out‚1219.27880859375 disk.sdb.in‚5.40625' proc.memory.availableMB‚ 9947.640625) disk./.usagePercent‚83.14385992861084% disk./.usageGB‚2973.6271591186523 disk.sda.in‚ 5.56640625 network.sent‚ 2.4870326e+07# disk.nvme0n1.in‚329983.51171875# disk.nvme0n1.out‚65493.24609375 disk.rbd0.out‚ 3.087890625 disk.sda1.in‚0 disk.md0.in‚ 10.97265625" disk.nvme1n1.in‚201421.6328125% memory_percent‚39.284420013427734) proc.memory.percent‚18.50123405456543 disk.md0p1.in‚ 10.97265625 disk.md0p1.out‚ 1174.99609375" proc.memory.rssMB‚ 3031.2421875²Á© Ê'‚(:… ¦íçԾ† gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚116.916' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.229 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%~Õ>~Ë'‚(:ó ¦íçԩ숇 disk.rbd0.out‚ 3.087890625 disk.sdb.out‚1221.2333984375 disk.sda.out‚1221.2333984375) proc.memory.availableMB‚ 9947.65234375" proc.memory.rssMB‚ 3031.2421875) disk./.usagePercent‚83.14964327598354 disk.rbd2.in‚0 disk.rbd2.out‚ 3.025390625 disk.sdb1.out‚0 disk.sda.in‚ 5.56640625 disk.dm-0.in‚531405.14453125! disk.dm-0.out‚133888.68359375 disk.sda1.out‚0 disk.md0.in‚ 10.97265625# disk.nvme0n1.out‚65586.51171875 disk.sda2.in‚ 5.56640625 disk.sda1.in‚0 disk.md0p1.in‚ 10.97265625 disk.md0p1.out‚1176.875 cpu‚25.187207297564076! disk.sdb2.out‚1221.2333984375" disk.nvme1n1.in‚201421.6328125 network.sent‚ 2.4969061e+07$ disk./.usageGB‚2973.833999633789 disk.rbd0.in‚0.015625 disk.sdb.in‚5.40625 disk.sdb1.in‚0% memory_percent‚39.284348487854004) proc.memory.percent‚18.50123405456543# disk.nvme0n1.in‚329983.51171875 disk.sdb2.in‚5.40625! disk.nvme1n1.out‚ 68302.171875 proc.cpu.threads‚17! disk.sda2.out‚1221.2333984375 network.recv‚ 3.810421e+06 disk.md0.out‚1176.875‹ ØœÌ'‚(:‘ µíçÔþ«ù… gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚118.474' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.618500000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ï'uÍ'‚(:ê µíçÔ³ú‡ disk.md0p1.in‚ 10.97265625 disk.rbd2.in‚0 disk.rbd2.out‚ 3.025390625" disk.nvme1n1.in‚201421.6328125# disk.nvme0n1.out‚65679.25390625 proc.cpu.threads‚17 disk.sdb2.in‚5.40625# disk.nvme1n1.out‚68350.10546875 disk.sda1.out‚0) disk./.usagePercent‚83.15318771212647 disk.sdb2.out‚ 1222.578125 disk.md0.in‚ 10.97265625 disk.md0.out‚ 1178.1484375 disk.rbd0.out‚ 3.087890625 disk.sdb1.in‚0 disk.sda.in‚ 5.56640625# disk.nvme0n1.in‚329983.51171875 disk.sdb.in‚5.40625 disk.sdb.out‚ 1222.578125 disk.sda1.in‚0 disk.sda2.out‚ 1222.578125 network.sent‚ 2.4979304e+07 disk.md0p1.out‚ 1178.1484375 disk.rbd0.in‚0.015625 disk.sda.out‚ 1222.578125 disk.sda2.in‚ 5.56640625" proc.memory.rssMB‚ 3031.2421875) proc.memory.percent‚18.50123405456543 disk.dm-0.out‚ 134029.359375% memory_percent‚39.290475845336914 cpu‚25.186977050930228 network.recv‚ 3.828063e+06$ disk./.usageGB‚2973.960765838623 disk.sdb1.out‚0 disk.dm-0.in‚531405.14453125( proc.memory.availableMB‚ 9946.6484375JWÛn4Î'/ _wandbruntime‚11121  _runtime‚11121 Š¦PÊÏ'Ä trainer/global_step‚9449 % _timestamp‚1.7884750683030229e+09 _runtime‚11121.487913919 _step‚773 ' lr-AdamW/pg1‚0.00011414556628852184 ' lr-AdamW/pg2‚0.00011414556628852184\û^@ÏÐ'É ' lr-AdamW/pg2‚0.00011414556628852184 trainer/global_step‚9449 % _timestamp‚1.7884750683030229e+09 _runtime‚11121.487913919 _step‚773 ' lr-AdamW/pg1‚0.00011414556628852184…VF‡4Ñ'/ _wandbruntime‚11121  _runtime‚11121T» ð'Ò'êO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚671 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚403300533138432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18770787418112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚68160397 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚426408036938752 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18771021086720 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚426407720282112 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 107239382 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚673 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18770787418112 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 179494703 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 256829075 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 175399779 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 436323778 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚384665100573432 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18717582466636 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚403300533138432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 179494703 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 179494701 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 trainer/global_step‚9449 $ _timestamp‚1.788475068303766e+09 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 175399106 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68159726 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚384664854552988 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19087863246336 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚365947289125712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18771021086720 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚403300784809984 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚384529745720320 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚365947272086352 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚774 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚384529763723264 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 107239380 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19087570529792 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 256829075 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 256802731 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 256802731 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚407320149752320 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚403300784809984 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚407320173692416 _runtime‚11121.488778563 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 436297432 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚384529763723264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18717811447720 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚384529745720320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582KøþÆÓ'ïO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚384529763723264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18717811447720 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚384529745720320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚671 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚403300533138432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18770787418112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚68160397 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚426408036938752 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18771021086720 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚426407720282112 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 107239382 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚673 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18770787418112 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 179494703 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 256829075 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 175399779 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 436323778 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚384665100573432 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18717582466636 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚403300533138432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 179494703 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 179494701 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 trainer/global_step‚9449 $ _timestamp‚1.788475068303766e+09 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 175399106 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68159726 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚384664854552988 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19087863246336 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚365947289125712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18771021086720 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚403300784809984 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚384529745720320 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚365947272086352 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚774 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚384529763723264 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 107239380 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19087570529792 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDevioL{/ceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 256829075 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 256802731 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 256802731 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚407320149752320 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚403300784809984 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚407320173692416 _runtime‚11121.488778563 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 436297432 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 256829423 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚384569049760256 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚384569031757312 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚426451398331904 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19089855215616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚403342034886144 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 175417985 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19089564609536 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 179512976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18772751443968 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18772985125888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 107250184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚384704343399398 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18719769936914 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 436368771 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚365984819483136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚679 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚426451712878080 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚68167801 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18772985125888 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚384569031757312 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚407361857662464 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18719540955622 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _step‚775 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚68167124 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚384569049760256 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18772751443968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚365984802443776 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 175417306 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 436342399 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 179512976 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚407361833722368 _runtime‚11122.487557724 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 179512978 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 256829423 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚403341783201280 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 436342399 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚677 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚403342034886144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 256855793 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 trainer/global_step‚9449 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 436368771 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0Î……ˆ'Ö'úM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 436342399 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 179512976 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚407361833722368 _runtime‚11122.487557724 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 179512978 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 256829423 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚403341783201280 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 436342399 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚677 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚403342034886144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 256855793 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 trainer/global_step‚9449 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 436368771 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 256829423 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚384569049760256 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚384569031757312 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚426451398331904 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19089855215616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚403342034886144 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 175417985 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19089564609536 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 179512976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18772751443968 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18772985125888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 107250184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚384704343399398 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18719769936914 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 436368771 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚365984819483136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚679 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚426451712878080 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚68167801 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18772985125888 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚384569031757312 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚407361857662464 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18719540955622 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _step‚775 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚68167124 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚384569049760256 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18772751443968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚365984802443776 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 175417306 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0‡¦õD´4×'/ _wandbruntime‚11122  _runtime‚11122—M0^Ø'Ø 1loss/FBt4/assignment_loss‚0.08995171752758235 1loss/SRqqt3/detection_loss‚0.7346935359019677 /loss/FBt1/detection_loss‚2.6792025107603807 2loss/SRqqt1/assignment_loss‚1.3452507246285677 /loss/FRt3/detection_loss‚1.3123817816376686 1loss/SRqqt2/detection_loss‚2.2156725293025374 trainer/global_step‚9449 2loss/SRqqt2/assignment_loss‚1.8079367543105036 2loss/SRqqt3/assignment_loss‚1.0032841232786118 1loss/FBt1/assignment_loss‚0.02816891245751007 /loss/FRt2/detection_loss‚1.9179069451727289 1loss/SRqqt1/detection_loss‚2.0444179616868494 /loss/FBt3/detection_loss‚0.4632434724994442 %loss/total_loss‚42.08184916326681 epoch‚7 _step‚776 0loss/FRt3/assignment_loss‚1.4082544581456617 1loss/SRqqt4/assignment_loss‚1.262699332939727 1loss/FBt2/assignment_loss‚0.07472097707342694 1loss/SRqqt4/detection_loss‚2.4275698023183003 .loss/FBt4/detection_loss‚2.841783620417118 % _timestamp‚1.7884750693039627e+09 /loss/FRt1/assignment_loss‚2.807298015903782 0loss/FBt3/assignment_loss‚0.1756408065964184 /loss/FBt2/detection_loss‚2.4957862267127404 0loss/FRt4/assignment_loss‚3.4672404669580006 _runtime‚11122.488653959 0loss/FRt2/assignment_loss‚3.3760335708068587 .loss/FRt1/detection_loss‚2.627781367785222 .loss/FRt4/detection_loss‚3.474929548445202Žà1cÙ'Ý 0loss/FRt4/assignment_loss‚3.4672404669580006 _runtime‚11122.488653959 0loss/FRt2/assignment_loss‚3.3760335708068587 .loss/FRt1/detection_loss‚2.627781367785222 .loss/FRt4/detection_loss‚3.474929548445202 1loss/FBt4/assignment_loss‚0.08995171752758235 1loss/SRqqt3/detection_loss‚0.7346935359019677 /loss/FBt1/detection_loss‚2.6792025107603807 2loss/SRqqt1/assignment_loss‚1.3452507246285677 /loss/FRt3/detection_loss‚1.3123817816376686 1loss/SRqqt2/detection_loss‚2.2156725293025374 trainer/global_step‚9449 2loss/SRqqt2/assignment_loss‚1.8079367543105036 2loss/SRqqt3/assignment_loss‚1.0032841232786118 1loss/FBt1/assignment_loss‚0.02816891245751007 /loss/FRt2/detection_loss‚1.9179069451727289 1loss/SRqqt1/detection_loss‚2.0444179616868494 /loss/FBt3/detection_loss‚0.4632434724994442 %loss/total_loss‚42.08184916326681 epoch‚7 _step‚776 0loss/FRt3/assignment_loss‚1.4082544581456617 1loss/SRqqt4/assignment_loss‚1.262699332939727 1loss/FBt2/assignment_loss‚0.07472097707342694 1loss/SRqqt4/detection_loss‚2.4275698023183003 .loss/FBt4/detection_loss‚2.841783620417118 % _timestamp‚1.7884750693039627e+09 /loss/FRt1/assignment_loss‚2.807298015903782 0loss/FBt3/assignment_loss‚0.1756408065964184 /loss/FBt2/detection_loss‚2.4957862267127404ˆôåGÐÚ'‚(:… ÄíçÔµ¥á† gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚95.806' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚23.9515 gpu.0.smClock‚1260 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*—d—ŒÛ'‚(:  ÄíçÔÑÕ‘‡ network.recv‚ 3.832129e+06 disk.sdb2.in‚5.40625 disk.sdb1.in‚0 disk.sda1.in‚0 disk.rbd0.in‚0.015625 disk.sda.in‚ 5.56640625# disk.nvme0n1.in‚329983.51171875 disk.md0p1.out‚ 1179.90234375 network.sent‚ 2.4985127e+07" disk.nvme1n1.out‚ 68488.1328125% disk./.usageGB‚2974.0854454040527 disk.sdb.in‚5.40625! disk.dm-0.out‚134167.65234375% memory_percent‚39.287662506103516" disk.sdb2.out‚1224.40673828125 disk.rbd2.out‚ 3.025390625" disk.nvme1n1.in‚201421.6328125! disk.sdb.out‚1224.40673828125! disk.sda.out‚1224.40673828125 disk.dm-0.in‚531405.14453125 disk.sda1.out‚0" disk.sda2.out‚1224.40673828125 disk.sda2.in‚ 5.56640625 disk.md0p1.in‚ 10.97265625' proc.memory.availableMB‚ 9947.109375) proc.memory.percent‚18.50123405456543 cpu‚25.186748664267334) disk./.usagePercent‚83.15667380495833 disk.md0.in‚ 10.97265625 disk.md0.out‚ 1179.90234375 disk.sdb1.out‚0# disk.nvme0n1.out‚65679.51953125" proc.memory.rssMB‚ 3031.2421875 disk.rbd0.out‚ 3.087890625 disk.rbd2.in‚0 proc.cpu.threads‚17ÛsÈ‘Ü'‚(:† ÓíçÔŒžÁ‡ gpu.0.gpu‚39.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚138.302' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.5755 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½Õ=­…Ý'‚(:ú ÓíçÔ´ Ì‡! disk.sdb2.out‚1226.2783203125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda.in‚ 5.56640625 disk.sda1.out‚0 network.sent‚ 2.5092211e+07 network.recv‚ 3.83879e+06 disk.sdb2.in‚5.40625 disk.rbd0.in‚0.015625( disk./.usagePercent‚83.1566754048663" disk.nvme1n1.in‚201421.6328125" disk.nvme0n1.out‚ 65679.5703125% memory_percent‚39.287638664245605 proc.cpu.threads‚17 disk.md0.out‚ 1181.7109375 disk.sdb.out‚1226.2783203125 disk.dm-0.in‚531405.14453125 disk.md0p1.out‚ 1181.7109375! disk.sda2.out‚1226.2783203125 disk.md0.in‚ 10.97265625 disk.rbd0.out‚ 3.087890625 disk.rbd2.in‚0 disk.sdb.in‚5.40625) proc.memory.availableMB‚ 9947.11328125" proc.memory.rssMB‚ 3031.2421875) proc.memory.percent‚18.50123405456543% disk./.usageGB‚2974.0855026245117! disk.dm-0.out‚134271.80078125 cpu‚25.18645119337363 disk.rbd2.out‚ 3.025390625# disk.nvme1n1.out‚68592.23046875 disk.sda.out‚1226.2783203125# disk.nvme0n1.in‚329983.51171875 disk.sda1.in‚0 disk.sda2.in‚ 5.56640625 disk.md0p1.in‚ 10.97265625®C^É‘Þ'‚(:† âíçÔ‘øȆ gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚138.642' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.6605 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨8BÔŠß'‚(:ÿ âíçÔäã݇ disk.sdb2.in‚5.40625) disk./.usagePercent‚83.15684051537004 disk.sda1.in‚0 disk.rbd0.out‚ 3.087890625# disk.nvme0n1.in‚329983.51171875 proc.cpu.threads‚17$ disk./.usageGB‚2974.091407775879 disk.md0p1.out‚ 1183.5390625 disk.sdb1.in‚0 network.sent‚ 2.5108526e+07! disk.dm-0.out‚134387.41015625 disk.md0.out‚ 1183.5390625! disk.nvme1n1.out‚ 68707.515625 disk.sda.in‚ 5.56640625) proc.memory.availableMB‚ 9946.84765625 network.recv‚ 3.862914e+06 disk.md0.in‚ 10.97265625 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 3.025390625! disk.sdb.out‚1228.15966796875" disk.sdb2.out‚1228.15966796875" disk.sda2.out‚1228.15966796875 disk.md0p1.in‚ 10.97265625 disk.rbd2.in‚0! disk.sda.out‚1228.15966796875 cpu‚25.186156105513728 disk.sda2.in‚ 5.56640625" disk.nvme1n1.in‚201421.6328125% memory_percent‚39.289259910583496) proc.memory.percent‚18.50123405456543 disk.dm-0.in‚531405.14453125 disk.sda1.out‚0 disk.sdb1.out‚0# disk.nvme0n1.out‚65679.89453125" proc.memory.rssMB‚ 3031.2421875 disk.sdb.in‚5.40625—PO4à'/ _wandbruntime‚11173  _runtime‚11173ÖuÊá'Ä trainer/global_step‚9499 % _timestamp‚1.7884751205829916e+09 _runtime‚11173.768053612 _step‚777 ' lr-AdamW/pg1‚0.00011006618342211815 ' lr-AdamW/pg2‚0.00011006618342211815¿9·ÐÏâ'É % _timestamp‚1.7884751205829916e+09 _runtime‚11173.768053612 _step‚777 ' lr-AdamW/pg1‚0.00011006618342211815 ' lr-AdamW/pg2‚0.00011006618342211815 trainer/global_step‚9499‰è…bÊ4ã'/ _wandbruntime‚11173  _runtime‚11173œAa÷á ä'êO UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 180408453 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18869003124736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚636 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18869236793344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚9499 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚428591759688704 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚405363302366208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 180408453 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _runtime‚11173.769261268 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 176289579 H?DeviceStatsMonitor.on_train_batch_start/inactive_splÿë*it.all.peak‚1319 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 258167027 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚405363050694656 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18869003124736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 176290215 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 $ _timestamp‚1.788475120583982e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚386494065572864 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚386494047569920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚409404372194816 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚367823806996912 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 438575480 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 258167027 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 107779482 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18815749488228 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚386494065572864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 258140683 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚428592065859584 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚386639556485140 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚405363050694656 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚386494047569920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚68510733 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚778 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 180408451 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 180408451 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚386639310464696 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚405363302366208 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚634 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19187411433984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚405363050694656 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18869003124736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 176290215 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 $ _timestamp‚1.788475120583982e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚386494065572864 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚386494047569920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚409404372194816 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚367823806996912 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 438575480 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 258167027 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 107779482 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18815749488228 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚386494065572864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 258140683 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚428592065859584 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚386639556485140 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚405363050694656 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚386494047569920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚68510733 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚778 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 180408451 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 180408451 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚386639310464696 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚405363302366208 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚634 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19187411433984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18869003124736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚636 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18869236793344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚9499 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚428591759688704 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚405363302366208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 180408453 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _runtime‚11173.769261268 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 176289579 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 258167027 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚405404300757504 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 180426726 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚405404552442368 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚367861320314976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 180426728 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 438620473 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 258167375 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 176307822 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18817478996130 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚409446032224768 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚386679045331758 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚386678799311106 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚409446056164864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18870967150592 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18871200832512 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚428635435743232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _runtime‚11174.766036105 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 107790282 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 176307128 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18817707977422 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 % _timestamp‚1.7884751215802786e+09 _step‚779 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18870967150592 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚428635748192256 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19189692027392 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚386533333606912 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 438594101 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 180426726 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚386533333606912 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚694 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚692 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚405404552442368 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚367861320314976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 180426728 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 438620473 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 258167375 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 176307822 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18817478996130 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚409446032224768 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚386679045331758 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚386678799311106 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚409446056164864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18870967150592 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18871200832512 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚428635435743232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 WIDeviceStaOƒ>–VtsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _runtime‚11174.766036105 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 107790282 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 176307128 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18817707977422 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 % _timestamp‚1.7884751215802786e+09 _step‚779 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18870967150592 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚428635748192256 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19189692027392 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚386533333606912 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 438594101 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 180426726 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚386533333606912 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚694 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚692 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚405404300757504 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 180426726 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496‹©ô¯•4é'/ _wandbruntime‚11174  _runtime‚11174åÕìsZê'Ô /loss/FRt1/assignment_loss‚2.878692533288683 1loss/FBt2/assignment_loss‚0.20662331160430886 1loss/SRqqt2/detection_loss‚2.2928602150801956 .loss/FBt2/detection_loss‚2.285216884179549 /loss/FRt2/assignment_loss‚2.856584287003467 1loss/SRqqt4/assignment_loss‚1.136711450992152 -loss/FRt2/detection_loss‚2.19250040965792 _step‚780 1loss/FBt4/assignment_loss‚0.12960297143500554 /loss/FRt4/detection_loss‚2.6759795596202216 1loss/SRqqt1/detection_loss‚1.7796453908085823 epoch‚7 _runtime‚11174.767371181 0loss/FRt3/assignment_loss‚1.3465319335460664 1loss/SRqqt2/assignment_loss‚1.318761207163334 .loss/FRt1/detection_loss‚2.582372409956796 1loss/SRqqt4/detection_loss‚2.2276925798505545 /loss/FBt3/detection_loss‚0.5048762485976257 2loss/SRqqt3/assignment_loss‚0.8732815880635844 /loss/FBt1/detection_loss‚1.9923157840967178 &loss/total_loss‚38.445299393593174 /loss/FRt4/assignment_loss‚2.659401045905219 /loss/FBt4/detection_loss‚2.6225289702415466 trainer/global_step‚9499 1loss/SRqqt1/assignment_loss‚1.363077144731175 0loss/FBt1/assignment_loss‚0.1609107656256299 1loss/FBt3/assignment_loss‚0.06477590472539024 /loss/FRt3/detection_loss‚1.3327606148086488 1loss/SRqqt3/detection_loss‚0.9615961826108035 % _timestamp‚1.7884751215824513e+09µšà_ë'Ù 0loss/FRt3/assignment_loss‚1.3465319335460664 1loss/SRqqt2/assignment_loss‚1.318761207163334 .loss/FRt1/detection_loss‚2.582372409956796 1loss/SRqqt4/detection_loss‚2.2276925798505545 /loss/FBt3/detection_loss‚0.5048762485976257 2loss/SRqqt3/assignment_loss‚0.8732815880635844 /loss/FBt1/detection_loss‚1.9923157840967178 &loss/total_loss‚38.445299393593174 /loss/FRt4/assignment_loss‚2.659401045905219 /loss/FBt4/detection_loss‚2.6225289702415466 trainer/global_step‚9499 1loss/SRqqt1/assignment_loss‚1.363077144731175 0loss/FBt1/assignment_loss‚0.1609107656256299 1loss/FBt3/assignment_loss‚0.06477590472539024 /loss/FRt3/detection_loss‚1.3327606148086488 1loss/SRqqt3/detection_loss‚0.9615961826108035 % _timestamp‚1.7884751215824513e+09 /loss/FRt1/assignment_loss‚2.878692533288683 1loss/FBt2/assignment_loss‚0.20662331160430886 1loss/SRqqt2/detection_loss‚2.2928602150801956 .loss/FBt2/detection_loss‚2.285216884179549 /loss/FRt2/assignment_loss‚2.856584287003467 1loss/SRqqt4/assignment_loss‚1.136711450992152 -loss/FRt2/detection_loss‚2.19250040965792 _step‚780 1loss/FBt4/assignment_loss‚0.12960297143500554 /loss/FRt4/detection_loss‚2.6759795596202216 1loss/SRqqt1/detection_loss‚1.7796453908085823 epoch‚7 _runtime‚11174.767371181ŒÙÙµx’ì'‚(:‡ ñíçÔµŠŸ‡ gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚117.525' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.38125 gpu.0.smClock‚1260 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G‘E8‡í'‚(:ü ñíçÔÂá‡! disk.dm-0.out‚134393.59765625 disk.md0p1.in‚ 10.97265625 disk.rbd0.out‚ 3.087890625 network.sent‚ 2.5114187e+07 disk.sda1.in‚0 network.recv‚ 3.866569e+06 disk.sda2.in‚ 5.56640625( proc.memory.availableMB‚ 9946.7890625" disk.sdb2.out‚1229.95654296875" disk.nvme1n1.in‚201421.6328125! disk.sdb.out‚1229.95654296875! disk.sda.out‚1229.95654296875 disk.sdb2.in‚5.40625 disk.rbd0.in‚0.015625$ memory_percent‚39.28961753845215) proc.memory.percent‚18.50123405456543% disk./.usageGB‚2973.8142700195312 disk.sdb.in‚5.40625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda.in‚ 5.56640625" disk.sda2.out‚1229.95654296875" proc.memory.rssMB‚ 3031.2421875 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sda1.out‚0 disk.md0.out‚ 1185.26171875 cpu‚25.18588452155336 disk.rbd2.out‚ 3.025390625 disk.md0p1.out‚ 1185.26171875 disk.md0.in‚ 10.97265625 disk.nvme1n1.out‚ 68707.625# disk.nvme0n1.in‚329983.51171875# disk.nvme0n1.out‚65685.97265625 disk.dm-0.in‚531405.14453125) disk./.usagePercent‚83.14909162771139¢'[Õî'‚(:… €îçÔƒ‡Ê† gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚134.908' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.727 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ö“àÈuï'‚(:ê €îçÔÝ€â‡% disk./.usageGB‚2973.9644355773926 disk.rbd2.in‚0) proc.memory.percent‚18.50123405456543 cpu‚25.18565831222718 disk.sdb2.out‚ 1231.2890625 disk.md0.out‚ 1186.546875 disk.sdb1.out‚0" proc.memory.rssMB‚ 3031.2421875" disk.nvme1n1.in‚201421.6328125 disk.sda2.in‚ 5.56640625 network.sent‚ 2.5221367e+07 network.recv‚ 3.873414e+06 disk.sda1.out‚0 disk.sdb2.in‚5.40625 disk.md0.in‚ 10.97265625 disk.rbd0.out‚ 3.087890625# disk.nvme0n1.out‚65758.10546875 disk.sda1.in‚0 disk.sdb.out‚ 1231.2890625$ memory_percent‚39.28816318511963) proc.memory.availableMB‚ 9947.02734375 proc.cpu.threads‚17 disk.sda.in‚ 5.56640625 disk.md0p1.in‚ 10.97265625 disk.md0p1.out‚ 1186.546875 disk.rbd0.in‚0.015625# disk.nvme1n1.out‚68813.70703125 disk.sdb.in‚5.40625 disk.sda2.out‚ 1231.2890625) disk./.usagePercent‚83.15329031955837 disk.sda.out‚ 1231.2890625# disk.nvme0n1.in‚329983.51171875 disk.dm-0.out‚ 134571.8125 disk.dm-0.in‚531405.14453125 disk.rbd2.out‚ 3.025390625 disk.sdb1.in‚0OÕ§Ù’ð'‚(:‡ îçÔ¦¹Å‡ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚105.443' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚26.36075 gpu.0.smClock‚1200 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô8…«€ñ'‚(:õ îçÔįÿˆ disk.sdb.out‚1233.509765625 disk.sdb1.in‚0" disk.nvme0n1.out‚ 65758.2421875 disk.sda1.in‚0 network.recv‚ 3.877048e+06 disk.rbd2.in‚0 disk.sdb.in‚5.40625 disk.sda.in‚ 5.56640625! disk.dm-0.out‚134596.94921875 disk.sda2.in‚ 5.56640625 network.sent‚ 2.522891e+07 disk.sdb2.out‚1233.509765625 disk.md0.in‚ 10.97265625 disk.rbd2.out‚ 3.025390625 disk.sda.out‚1233.509765625 disk.md0p1.out‚ 1188.6953125 disk.md0.out‚ 1188.6953125 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 3.087890625 disk.sdb2.in‚5.40625 disk.dm-0.in‚531405.14453125) proc.memory.availableMB‚ 9947.27734375) disk./.usagePercent‚83.15772601110771 disk.sda2.out‚1233.509765625 disk.md0p1.in‚ 10.97265625" proc.memory.rssMB‚ 3031.2421875" disk.nvme1n1.in‚201421.6328125$ memory_percent‚39.28663730621338 cpu‚25.18538251002652 proc.cpu.threads‚17# disk.nvme1n1.out‚68838.70703125 disk.sdb1.out‚0# disk.nvme0n1.in‚329983.51171875 disk.sda1.out‚0) proc.memory.percent‚18.50123405456543$ disk./.usageGB‚2974.123077392578“Tï’œò'‚(:‘ žîçÔ¢¦‹‹ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚119.978' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.994499999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03T¶ƒó'‚(:ø žîçÔ“«) proc.memory.availableMB‚ 9947.03515625$ disk./.usageGB‚2974.341770172119 disk.rbd0.in‚0.015625" disk.nvme1n1.in‚201421.6328125 disk.sdb.in‚5.40625# disk.nvme0n1.in‚329983.51171875 disk.dm-0.out‚ 134833.515625 disk.md0p1.out‚ 1190.12890625 disk.md0.in‚ 10.97265625 disk.sdb1.in‚0 disk.sda1.in‚0 disk.sda2.in‚ 5.56640625 cpu‚25.185083898713447) proc.memory.percent‚18.50123405456543 disk.sdb2.out‚1235.001953125 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.dm-0.in‚531405.14453125 disk.sda2.out‚1235.001953125 disk.sdb.out‚1235.001953125# disk.nvme0n1.out‚65902.55078125" proc.memory.rssMB‚ 3031.2421875 network.recv‚ 3.881623e+06 disk.sdb2.in‚5.40625 disk.md0.out‚ 1190.12890625# disk.nvme1n1.out‚68930.96484375 disk.sda.in‚ 5.56640625$ memory_percent‚39.28811550140381 proc.cpu.threads‚17 network.sent‚ 2.5233086e+07) disk./.usagePercent‚83.16384075275371 disk.sda.out‚1235.001953125 disk.sda1.out‚0 disk.md0p1.in‚ 10.97265625 disk.rbd0.out‚ 3.087890625 disk.rbd2.out‚ 3.025390625å‚4ô'/ _wandbruntime‚11231  _runtime‚11231ðî¨Êõ'Ä _step‚781 ' lr-AdamW/pg1‚0.00010604977028176437 ' lr-AdamW/pg2‚0.00010604977028176437 trainer/global_step‚9549 % _timestamp‚1.7884751786838555e+09 _runtime‚11231.868790702ÃèÞ“Ïö'É ' lr-AdamW/pg2‚0.00010604977028176437 trainer/global_step‚9549 % _timestamp‚1.7884751786838555e+09 _runtime‚11231.868790702 _step‚781 ' lr-AdamW/pg1‚0.00010604977028176437ì÷+4÷'/ _wandbruntime‚11231  _runtime‚11231vÓ.%ñ'ø'ëO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚388458349419520 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 % _timestamp‚1.7884751786849227e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 108319582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚407425819922432 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18967452499968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19287554936832 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18913687528736 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚407425568250880 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68859133 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚369700324868112 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 259478635 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 181322203 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _runtime‚11231.869918347 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18967218831360 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 177179409 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚407425568250880 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚694 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚369700307828752 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _step‚782 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 440827182 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18967452499968 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚388458367422464 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚407425819922432 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚430776125634048 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚411488546757120 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19287262220288 trainer/global_step‚9549 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 259504979 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 108319580 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 259478635 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚696 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18967218831360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0y¦cZþù'ðO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚430775808977408 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 259504979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18913458547652 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚388458349419520 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚388458367422464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚388613766376404 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 181322203 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 177178713 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚411488570697216 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚388614012396848 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚407425568250880 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚694 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚369700307828752 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _step‚782 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 440827182 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18967452499968 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚388458367422464 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚407425819922432 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚430776125634048 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚411488546757120 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19287262220288 trainer/global_step‚9549 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 259504979 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 108319580 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 259478635 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚696 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18967218831360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚388458349419520 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 % _timestamp‚1.7884751786849227e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 108319582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚407425819922432 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18967452499968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19287554936832 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_al¬eºsø locations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18913687528736 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚407425568250880 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68859133 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚369700324868112 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 259478635 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 181322203 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _runtime‚11231.869918347 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18967218831360 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 177179409 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚388497653459456 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18969182857216 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚698 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 trainer/global_step‚9549 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 177196680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚430819507578368 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 440872175 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 440872175 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚407467069998592 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚369737855225536 _step‚783 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 259531697 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚411530254667264 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19289565360128 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 259505327 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18969416539136 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 259531697 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18915646017930 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 440845803 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 440845803 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19289276851200 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18915417036638 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 108330384 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚388497653459456 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 181340476 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309M?v'ü'úM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚388497653459456 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18969182857216 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚698 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 trainer/global_step‚9549 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 177196680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚430819507578368 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 440872175 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 440872175 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚407467069998592 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚369737855225536 _step‚783 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 259531697 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚411530254667264 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19289565360128 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 259505327 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18969416539136 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 259531697 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18915646017930 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 440845803 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 440845803 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19289276851200 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18915417036638 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 108330384 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚388497653459456 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 181340476 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚388497635456512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7884751799957557e+09 _runtime‚11233.181431367 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚411530230727168 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚407467069998592 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 259505327 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚388653501243466 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006õ¯¥4ý'/ _wandbruntime‚11233  _runtime‚11233ÜǾZþ'Ô /loss/FRt3/assignment_loss‚1.382073340592561 _runtime‚11233.183253175 0loss/FRt4/assignment_loss‚2.8357222931725636 2loss/SRqqt1/assignment_loss‚1.3504341841778822 1loss/FBt2/assignment_loss‚0.11745326729297453 0loss/SRqqt1/detection_loss‚2.128466673195362 1loss/SRqqt3/detection_loss‚0.7309994914559359 0loss/FRt1/assignment_loss‚3.2892168276011944 2loss/SRqqt4/assignment_loss‚1.7080895521424033 0loss/FBt1/assignment_loss‚0.5592994038713148 /loss/FRt3/detection_loss‚1.2287498264585013 -loss/FBt2/detection_loss‚2.02743582542126 % _timestamp‚1.7884751799985523e+09 2loss/SRqqt2/assignment_loss‚1.6940981062750022 -loss/FRt2/detection_loss‚2.13097492179116 /loss/FBt4/detection_loss‚2.6114310324192047 epoch‚7 _step‚784 /loss/FRt2/assignment_loss‚2.876704261109636 2loss/SRqqt3/assignment_loss‚0.8851927984793929 /loss/FRt4/detection_loss‚2.9933542800801143 /loss/FBt3/detection_loss‚0.3902624644882356 trainer/global_step‚9549 1loss/FBt4/assignment_loss‚0.08873072168455616 /loss/FRt1/detection_loss‚2.8519630897790194 0loss/SRqqt2/detection_loss‚2.366178716222445 1loss/FBt3/assignment_loss‚0.06562984979714957 /loss/FBt1/detection_loss‚2.2205199241638183 0loss/SRqqt4/detection_loss‚2.168216069539388 %loss/total_loss‚40.70119692121107¾Niª_ÿ'Ù 1loss/FBt3/assignment_loss‚0.06562984979714957 /loss/FBt1/detection_loss‚2.2205199241638183 0loss/SRqqt4/detection_loss‚2.168216069539388 %loss/total_loss‚40.70119692121107 /loss/FRt3/assignment_loss‚1.382073340592561 _runtime‚11233.183253175 0loss/FRt4/assignment_loss‚2.8357222931725636 2loss/SRqqt1/assignment_loss‚1.3504341841778822 1loss/FBt2/assignment_loss‚0.11745326729297453 0loss/SRqqt1/detection_loss‚2.128466673195362 1loss/SRqqt3/detection_loss‚0.7309994914559359 0loss/FRt1/assignment_loss‚3.2892168276011944 2loss/SRqqt4/assignment_loss‚1.7080895521424033 0loss/FBt1/assignment_loss‚0.5592994038713148 /loss/FRt3/detection_loss‚1.2287498264585013 -loss/FBt2/detection_loss‚2.02743582542126 % _timestamp‚1.7884751799985523e+09 2loss/SRqqt2/assignment_loss‚1.6940981062750022 -loss/FRt2/detection_loss‚2.13097492179116 /loss/FBt4/detection_loss‚2.6114310324192047 epoch‚7 _step‚784 /loss/FRt2/assignment_loss‚2.876704261109636 2loss/SRqqt3/assignment_loss‚0.8851927984793929 /loss/FRt4/detection_loss‚2.9933542800801143 /loss/FBt3/detection_loss‚0.3902624644882356 trainer/global_step‚9549 1loss/FBt4/assignment_loss‚0.08873072168455616 /loss/FRt1/detection_loss‚2.8519630897790194 0loss/SRqqt2/detection_loss‚2.366178716222445øD·ô’€(‚(:‡ ­îçÔ«‡ºˆ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚129.947' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.48675 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“0Z(‚(:ô ­îçÔ¨‡ò‰ disk.sdb1.out‚0 disk.sda.in‚ 5.56640625# disk.nvme0n1.in‚329983.51171875 disk.sda2.in‚ 5.56640625 disk.rbd0.in‚0.015625" proc.memory.rssMB‚ 3031.2421875) proc.memory.percent‚18.50123405456543 cpu‚25.184818434401826 network.sent‚ 2.523891e+07% disk./.usageGB‚2974.5913124084473 disk.dm-0.in‚531405.14453125" disk.nvme1n1.in‚201421.6328125" memory_percent‚39.288330078125' disk./.usagePercent‚83.170818058123" disk.sdb2.out‚1236.49951171875 disk.md0.in‚ 10.97265625# disk.nvme1n1.out‚68975.07421875 proc.cpu.threads‚17! disk.sda.out‚1236.49951171875 disk.md0p1.out‚ 1191.55859375 disk.rbd2.out‚ 3.025390625 proc.memory.availableMB‚9947 network.recv‚ 3.886224e+06! disk.sdb.out‚1236.49951171875 disk.sda1.out‚0" disk.sda2.out‚1236.49951171875 disk.sda1.in‚0 disk.rbd2.in‚0 disk.sdb.in‚5.40625# disk.nvme0n1.out‚65974.60546875 disk.dm-0.out‚134949.6796875 disk.md0p1.in‚ 10.97265625 disk.md0.out‚ 1191.55859375 disk.sdb1.in‚0 disk.sdb2.in‚5.40625 disk.rbd0.out‚ 3.0878906254­n’‚(‚(:‡ ¼îçÔžþà‡ gpu.0.gpu‚38.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚103.267' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.81675 gpu.0.smClock‚1185 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÌïÐlƒ(‚(:ô ¼îçÔšŒ‡‰ disk.sda1.in‚0 disk.dm-0.in‚531405.14453125" disk.nvme1n1.out‚ 69046.1953125' disk./.usagePercent‚83.170818058123" disk.nvme0n1.out‚ 66046.8828125 disk.dm-0.out‚ 135093.078125 disk.sda1.out‚0 disk.md0p1.out‚ 1193.38671875 disk.rbd2.out‚ 3.025390625 disk.sdb.in‚5.40625 cpu‚25.18450992145099 disk.sda2.in‚ 5.56640625 disk.rbd2.in‚0$ memory_percent‚39.28835391998291 proc.cpu.threads‚17 disk.sdb2.out‚1238.373046875 disk.rbd0.in‚0.015625" disk.nvme1n1.in‚201421.6328125) proc.memory.availableMB‚ 9946.99609375" proc.memory.rssMB‚ 3031.2421875 network.recv‚ 3.893439e+06 disk.sda2.out‚1238.373046875 disk.md0p1.in‚ 10.97265625 disk.sdb2.in‚5.40625 disk.sdb1.in‚0 disk.sda.in‚ 5.56640625 disk.sdb1.out‚0% disk./.usageGB‚2974.5913124084473# disk.nvme0n1.in‚329983.51171875 disk.md0.out‚ 1193.38671875 disk.sda.out‚1238.373046875 network.sent‚ 2.5349206e+07 disk.md0.in‚ 10.97265625 disk.rbd0.out‚ 3.087890625 disk.sdb.out‚1238.373046875) proc.memory.percent‚18.50123405456543²øM‘„(‚(:† ËîçÔűȇ gpu.0.gpu‚10.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚92.689' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚23.17225 gpu.0.smClock‚1215 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@pw´……(‚(:ú ËîçÔ—‰­ˆ disk.dm-0.in‚531405.14453125! disk.dm-0.out‚135348.79296875 disk.sda1.in‚0" proc.memory.rssMB‚ 3031.2421875 disk.md0.out‚ 1195.0390625( disk./.usagePercent‚83.1630736502067% disk./.usageGB‚2974.3143348693848 disk.sda.out‚1240.0810546875) proc.memory.percent‚18.50123405456543 cpu‚25.184268220750447 disk.md0p1.out‚ 1195.0390625" disk.nvme1n1.in‚201421.6328125 disk.sdb.in‚5.40625# disk.nvme0n1.in‚329983.51171875$ memory_percent‚39.28685188293457 disk.sdb.out‚1240.0810546875" disk.nvme0n1.out‚ 66118.9296875 disk.sdb1.in‚0 disk.sda2.in‚ 5.56640625 disk.md0.in‚ 10.97265625 disk.rbd0.out‚ 3.087890625 disk.rbd2.out‚ 3.025390625 disk.sda.in‚ 5.56640625 disk.sda1.out‚0( proc.memory.availableMB‚ 9947.2421875 proc.cpu.threads‚17 disk.rbd0.in‚0.015625! disk.sda2.out‚1240.0810546875 disk.md0p1.in‚ 10.97265625 disk.sdb2.in‚5.40625! disk.sdb2.out‚1240.0810546875 disk.rbd2.in‚0 network.sent‚ 2.5354485e+07 network.recv‚ 3.897326e+06# disk.nvme1n1.out‚69229.86328125 disk.sdb1.out‚0È õJ’†(‚(:‡ ÚîçÔ÷ Å‡ gpu.0.gpu‚24.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚144.821' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.20525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­ø¤A‡‡(‚(:ü ÚîçÔÁ©éˆ disk.md0.in‚ 10.97265625 disk.rbd0.out‚ 3.087890625 disk.rbd2.in‚0 disk.sdb1.out‚0! disk.sda.out‚1241.43310546875" proc.memory.rssMB‚ 3031.2421875 network.recv‚ 3.902041e+06) disk./.usagePercent‚83.17598960068344 disk.rbd0.in‚0.015625 disk.sda.in‚ 5.56640625# disk.nvme0n1.in‚329983.51171875$ memory_percent‚39.28985595703125 disk.sdb2.in‚5.40625! disk.sdb.out‚1241.43310546875" disk.nvme1n1.in‚201421.6328125# disk.nvme1n1.out‚69269.10546875 disk.sdb1.in‚0 disk.dm-0.in‚531405.14453125 disk.dm-0.out‚135460.1796875) proc.memory.percent‚18.50123405456543 disk.md0p1.out‚ 1196.33203125" disk.sdb2.out‚1241.43310546875 disk.rbd2.out‚ 3.025390625# disk.nvme0n1.out‚66191.07421875 disk.sda1.in‚0 disk.sda1.out‚0# proc.memory.availableMB‚9946.75 cpu‚25.184043835542255 disk.sda2.in‚ 5.56640625" disk.sda2.out‚1241.43310546875 proc.cpu.threads‚17 disk.md0.out‚ 1196.33203125 disk.sdb.in‚5.40625 network.sent‚ 2.5358661e+07 disk.md0p1.in‚ 10.97265625% disk./.usageGB‚2974.7762718200684»=©/4ˆ(/ _wandbruntime‚11291  _runtime‚11291ÞT(pɉ(à _runtime‚11291.96512265 _step‚785 ' lr-AdamW/pg1‚0.00010209716388193879 ' lr-AdamW/pg2‚0.00010209716388193879 trainer/global_step‚9599 % _timestamp‚1.7884752387800877e+09Íx1÷Ί(È _step‚785 ' lr-AdamW/pg1‚0.00010209716388193879 ' lr-AdamW/pg2‚0.00010209716388193879 trainer/global_step‚9599 % _timestamp‚1.7884752387800877e+09 _runtime‚11291.96512265‘O/›ª4‹(/ _wandbruntime‚11291  _runtime‚11291§AÈŒ(ëO UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 260842931 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19011625569244 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 _step‚786 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor4Î>fá%.on_train_batch_start/inactive_split.all.freed‚ 178070502 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚390588468308556 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 260816587 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚390422651269120 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19065434537984 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚371576825699952 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19065668206592 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 182235953 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚432960115066880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚9599 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 260816587 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19387061539328 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚390422651269120 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 443078884 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 182235953 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚390588222288112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚371576842739312 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _runtime‚11291.966221545 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚413572769199616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚409488085807104 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 178071211 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 443052538 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19387345867264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 260842931 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736¼¹ö'(ðO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚409488337478656 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚390422669272064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚432959806798848 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚69211529 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884752387809162e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚390588222288112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚371576842739312 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _runtime‚11291.966221545 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚413572769199616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚409488085807104 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 178071211 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 443052538 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19387345867264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 260842931 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 260842931 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19011625569244 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 _step‚786 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 178070502 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚390588468308556 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 260816587 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚390422651269120 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19065434537984 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚371576825699952 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19065668206592 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 182235953 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚432960115066880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚9599 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 260816587 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19387061539328 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚390422651269120 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 443078884 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 182235953 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 260869649 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19067632245760 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚433003489463296 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 260843279 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 443097505 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 108870484 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 182254228 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 182254226 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 178088620 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19389338256896 trainer/global_step‚9599 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 182254226 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301963264 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚390461937306112 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚409529335869952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 260869649 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚69217464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 178087946 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚390627711134522 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚11293.055158014 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 443123877 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚371614356057376 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚413614429229568 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚390627957155174 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19067632245760 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚413614453169664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 278023168 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 _step‚787 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19389060233728 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0Àbö¡× (úM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 443097505 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚390461955309056 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚674 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚433003791426560 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 178087946 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚390627711134522 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚11293.055158014 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 443123877 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚371614356057376 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚413614429229568 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚390627957155174 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19067632245760 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚413614453169664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 278023168 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 _step‚787 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19389060233728 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚409529587554816 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 182254228 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19013584058438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚409529587554816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19067398563840 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 260869649 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19067632245760 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚433003489463296 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 260843279 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 443097505 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 108870484 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 182254228 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 182254226 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 178088620 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19389338256896 trainer/global_step‚9599 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 182254226 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301963264 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚390461937306112 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚409529335869952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 260869649 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚69217464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 183149701 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚392386971121664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 262180883 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 183149701 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚415656943761920 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19109563609752 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19163650244608 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚790 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚392386953118720 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 445304240 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19163883913216 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 _runtime‚11348.312238358 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 183149703 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚392386971121664 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚69560368 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚411550603363328 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 109399780 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚671 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 445330586 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚673 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚435143793795584 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚411550603363328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0ƒÃQ¡(ðO M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 183149701 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚392386971121664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 262180883 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 183149701 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚415656943761920 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19109563609752 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19163650244608 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚790 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚392386953118720 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 445304240 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19163883913216 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 _runtime‚11348.312238358 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 183149703 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚392386971121664 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚69560368 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚411550603363328 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 109399780 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚671 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 445330586 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚673 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚435143793795584 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚411550603363328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19163883913216 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582–ÝòóZ4¢(/ _wandbruntime‚11349  _runtime‚113498$Zù&£(óM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19489132939264 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚373490890967936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 178977789 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚411592105111040 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19111293117654 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 445349207 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚415698627731968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚392426257158656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 445375579 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 % _timestamp‚1.7884752962370672e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 262207601 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 183167976 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚392602413066882 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 262181231 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚392426239155712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 262181231 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚415698651672064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚435187784611328 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 178978471 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19111522098946 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚392426257158656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 _step‚791 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚9649 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 183167978 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19165847952384 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19165614270464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 183167978 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚435187474259456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚392602167046230 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _runtime‚ 11349.4226555 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚411591853426176 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19165847952384 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582ü_#þ&¤(øM >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚392426257158656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 _step‚791 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚9649 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 183167978 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19165847952384 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19165614270464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 183167978 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚435187474259456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚392602167046230 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _runtime‚ 11349.4226555 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚411591853426176 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19165847952384 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19489132939264 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚373490890967936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 178977789 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚411592105111040 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19111293117654 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 445349207 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚415698627731968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚392426257158656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 445375579 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 % _timestamp‚1.7884752962370672e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 262207601 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 183167976 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚392602413066882 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 262181231 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚392426239155712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 262181231 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚415698651672064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚435187784611328 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 178978471 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19111522098946—ç"ØA4¥(/ _wandbruntime‚11349  _runtime‚11349¿3‡IY¦(Ó trainer/global_step‚9649 _step‚792 0loss/FBt4/assignment_loss‚0.3827755929221587 1loss/SRqqt3/detection_loss‚0.8632152731679067 .loss/FBt1/detection_loss‚2.567877892936979 .loss/FRt4/assignment_loss‚2.92956791474269 1loss/SRqqt1/assignment_loss‚1.295572625266181 /loss/FRt3/detection_loss‚1.5316251878429092 1loss/SRqqt4/detection_loss‚2.0135723541180295 /loss/FRt2/detection_loss‚2.1402720317101656 .loss/FRt4/detection_loss‚2.655053559022072 epoch‚7 /loss/FBt2/assignment_loss‚0.324174210572739 .loss/FBt2/detection_loss‚2.036216127020972 /loss/FBt3/detection_loss‚0.4170745536484274 /loss/FBt4/detection_loss‚2.4033781106655416 %loss/total_loss‚41.54010788899833 0loss/FRt1/assignment_loss‚2.2219568292299905 0loss/FRt2/assignment_loss‚3.9179337092421274 2loss/SRqqt3/assignment_loss‚1.0124451921557247 2loss/SRqqt4/assignment_loss‚1.6032549142061423 % _timestamp‚1.7884752962389402e+09 _runtime‚11349.423633454 1loss/FBt3/assignment_loss‚0.06830735424310025 /loss/FRt1/detection_loss‚2.7665837506453195 0loss/SRqqt1/detection_loss‚2.048588612841235 0loss/FBt1/assignment_loss‚0.8048951354403105 0loss/FRt3/assignment_loss‚1.2803714248267086 2loss/SRqqt2/assignment_loss‚1.0865509659051895 1loss/SRqqt2/detection_loss‚3.1688445666257072Â9w^§(Ø /loss/FRt2/detection_loss‚2.1402720317101656 .loss/FRt4/detection_loss‚2.655053559022072 epoch‚7 /loss/FBt2/assignment_loss‚0.324174210572739 .loss/FBt2/detection_loss‚2.036216127020972 /loss/FBt3/detection_loss‚0.4170745536484274 /loss/FBt4/detection_loss‚2.4033781106655416 %loss/total_loss‚41.54010788899833 0loss/FRt1/assignment_loss‚2.2219568292299905 0loss/FRt2/assignment_loss‚3.9179337092421274 2loss/SRqqt3/assignment_loss‚1.0124451921557247 2loss/SRqqt4/assignment_loss‚1.6032549142061423 % _timestamp‚1.7884752962389402e+09 _runtime‚11349.423633454 1loss/FBt3/assignment_loss‚0.06830735424310025 /loss/FRt1/detection_loss‚2.7665837506453195 0loss/SRqqt1/detection_loss‚2.048588612841235 0loss/FBt1/assignment_loss‚0.8048951354403105 0loss/FRt3/assignment_loss‚1.2803714248267086 2loss/SRqqt2/assignment_loss‚1.0865509659051895 1loss/SRqqt2/detection_loss‚3.1688445666257072 trainer/global_step‚9649 _step‚792 0loss/FBt4/assignment_loss‚0.3827755929221587 1loss/SRqqt3/detection_loss‚0.8632152731679067 .loss/FBt1/detection_loss‚2.567877892936979 .loss/FRt4/assignment_loss‚2.92956791474269 1loss/SRqqt1/assignment_loss‚1.295572625266181 /loss/FRt3/detection_loss‚1.5316251878429092 1loss/SRqqt4/detection_loss‚2.0135723541180295˜ŠªCÔ¨(‚(:„ ¥ïçÔ™À‡ gpu.0.gpu‚8.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚110.396' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚27.599 gpu.0.smClock‚1230 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª:I錩(‚(:  ¥ïçÔúÕ¦‰" disk.sdb2.out‚1249.78759765625 disk.rbd0.in‚0.015625) proc.memory.availableMB‚ 9946.64453125! disk.dm-0.out‚136214.93359375 disk.rbd0.out‚ 3.087890625# disk.nvme1n1.out‚69591.25390625 disk.sdb1.in‚0" disk.nvme0n1.out‚ 66623.6796875" proc.memory.rssMB‚ 3031.2421875) disk./.usagePercent‚83.18480349375533# disk.nvme0n1.in‚329983.51171875 cpu‚25.1830490732144 network.recv‚ 3.925826e+06 disk.sdb.in‚5.40625! disk.sda.out‚1249.78759765625 disk.sda1.in‚0% memory_percent‚39.290499687194824 disk.md0p1.in‚ 10.97265625 disk.rbd2.out‚ 3.025390625 disk.dm-0.in‚531405.14453125) proc.memory.percent‚18.50123405456543 network.sent‚ 2.5502522e+07 disk.sda2.in‚ 5.56640625" disk.nvme1n1.in‚201421.6328125 disk.sdb1.out‚0 disk.sda1.out‚0" disk.sda2.out‚1249.78759765625 disk.md0p1.out‚ 1204.37109375 disk.sdb2.in‚5.40625 disk.md0.out‚ 1204.37109375 disk.rbd2.in‚0 disk.sda.in‚ 5.56640625 proc.cpu.threads‚17% disk./.usageGB‚2975.0914993286133 disk.md0.in‚ 10.97265625! disk.sdb.out‚1249.78759765625ÀÓŠ$ª(‚(:… ´ïçÔ¦§Á† gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚88.678' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚22.1695 gpu.0.smClock‚1185 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»÷·†«(‚(:û ´ïçÔÌúõ‡" disk.nvme1n1.in‚201421.6328125 disk.sdb1.in‚0# disk.nvme0n1.in‚329983.51171875 disk.dm-0.in‚531405.14453125 disk.md0p1.in‚ 10.97265625 disk.sdb1.out‚0 disk.sda.in‚ 5.56640625 network.recv‚ 3.93337e+06% disk./.usageGB‚2974.8163871765137 disk.md0.in‚ 10.97265625! disk.sda.out‚1251.30224609375" disk.sdb2.out‚1251.30224609375( proc.memory.availableMB‚ 9946.6484375 proc.cpu.threads‚17! disk.sdb.out‚1251.30224609375 disk.sdb2.in‚5.40625) proc.memory.percent‚18.50123405456543 disk.rbd2.out‚ 3.025390625 disk.sda1.out‚0" proc.memory.rssMB‚ 3031.2421875" disk.sda2.out‚1251.30224609375) disk./.usagePercent‚83.17711124283925 disk.rbd0.in‚0.015625 disk.rbd2.in‚0# disk.nvme1n1.out‚69775.23046875 disk.sdb.in‚5.40625# disk.nvme0n1.out‚66695.86328125 disk.sda1.in‚0 disk.md0.out‚ 1205.828125 disk.rbd0.out‚ 3.087890625 disk.sda2.in‚ 5.56640625 disk.md0p1.out‚ 1205.828125% memory_percent‚39.290475845336914 network.sent‚ 2.5599999e+07 disk.dm-0.out‚ 136471.09375 cpu‚25.182792765975396!Vsy‰¬(‚(:þ ÃïçÔüÍŸ‡% disk./.usageGB‚2974.7764282226562 disk.sda1.in‚0 disk.rbd0.in‚0.015625" disk.nvme1n1.in‚201421.6328125 disk.sdb1.in‚0% memory_percent‚39.288926124572754) proc.memory.percent‚18.50123405456543 disk.sda2.in‚ 5.56640625 disk.sdb2.in‚5.40625! disk.sdb2.out‚1252.8056640625! disk.dm-0.out‚136582.33203125 network.sent‚ 2.5605278e+07) disk./.usagePercent‚83.17599397376526 disk.md0.out‚ 1207.27734375 disk.rbd2.in‚0 disk.sdb1.out‚0" disk.nvme0n1.out‚ 66767.9140625 cpu‚25.18564283892871 network.recv‚ 3.937023e+06 disk.md0p1.out‚ 1207.27734375 disk.md0.in‚ 10.97265625# disk.nvme0n1.in‚329983.51171875 disk.sda1.out‚0 disk.sda.out‚1252.8056640625 disk.dm-0.in‚531405.14453125 disk.md0p1.in‚ 10.97265625 disk.rbd0.out‚ 3.087890625 disk.sdb.out‚1252.8056640625 disk.sda.in‚ 5.56640625" proc.memory.rssMB‚ 3031.2421875 disk.rbd2.out‚ 3.025390625# disk.nvme1n1.out‚69814.41796875 disk.sdb.in‚5.40625 proc.cpu.threads‚17! disk.sda2.out‚1252.8056640625) proc.memory.availableMB‚ 9946.90234375éÆé ­(‚(:… ÃïçÔµ­Â‡ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚85.893' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚21.47325 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e u½™®(‚(:  ÒïçÔ´Éׇ% memory_percent‚39.291977882385254) disk./.usagePercent‚83.17786970588315# disk.nvme1n1.out‚69881.51953125! disk.sda.out‚1254.66552734375! disk.dm-0.out‚136793.78515625 disk.md0p1.out‚ 1209.08984375) proc.memory.availableMB‚ 9946.40234375 disk.rbd2.out‚ 3.025390625" disk.nvme1n1.in‚201421.6328125 disk.sda.in‚ 5.56640625 cpu‚25.19051387930741 disk.md0.out‚ 1209.08984375 disk.rbd0.out‚ 3.087890625 disk.sdb.in‚5.40625 disk.md0p1.in‚ 10.97265625 network.recv‚ 3.941537e+06" disk.sdb2.out‚1254.66552734375 disk.rbd0.in‚0.015625" proc.memory.rssMB‚ 3031.2421875) •r¸‘óproc.memory.percent‚18.50123405456543 network.sent‚ 2.5611691e+07 disk.sdb2.in‚5.40625 disk.rbd2.in‚0 disk.dm-0.in‚531405.14453125" disk.sda2.out‚1254.66552734375% disk./.usageGB‚2974.8435134887695! disk.sdb.out‚1254.66552734375 disk.sdb1.in‚0# disk.nvme0n1.in‚329983.51171875 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0.in‚ 10.97265625! disk.nvme0n1.out‚ 66912.265625 disk.sda2.in‚ 5.56640625²ÝwM‘¯(‚(:† ÒïçÔř̇ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ù8$ž‘°(‚(:† áïçÔǑЇ gpu.0.gpu‚9.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚148.223' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚37.05575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æ¢ìŒ±(‚(:  áïçÔÓè‡" disk.sdb2.out‚1256.10205078125 proc.cpu.threads‚17% disk./.usageGB‚2974.9684104919434 disk.md0.in‚ 10.97265625! disk.sda.out‚1256.10205078125% memory_percent‚39.289140701293945 network.sent‚ 2.5616608e+07 disk.sda1.in‚0 disk.sdb1.in‚0 disk.sda.in‚ 5.56640625 disk.dm-0.in‚531405.14453125 disk.dm-0.out‚136934.8984375" disk.sda2.out‚1256.10205078125 disk.md0p1.out‚ 1210.45703125 disk.rbd0.in‚0.015625! disk.sdb.out‚1256.10205078125) proc.memory.percent‚18.50123405456543 disk.rbd2.in‚0# disk.nvme0n1.out‚66984.31640625" proc.memory.rssMB‚ 3031.2421875 cpu‚25.19474026695368 network.recv‚ 3.944911e+06 disk.sdb2.in‚5.40625 disk.rbd0.out‚ 3.087890625 disk.sdb1.out‚0( proc.memory.availableMB‚ 9946.8671875 disk.sdb.in‚5.40625# disk.nvme0n1.in‚329983.51171875) disk./.usagePercent‚83.18136187836535 disk.md0.out‚ 1210.45703125 disk.rbd2.out‚ 3.025390625" disk.nvme1n1.in‚201421.6328125# disk.nvme1n1.out‚69950.58203125 disk.sda1.out‚0 disk.sda2.in‚ 5.56640625 disk.md0p1.in‚ 10.97265625ò¨¹’²(‚(:‡ ðïçÔë¿ÿ† gpu.0.gpu‚45.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚136.743' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.18575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒÉ²†³(‚(:û ðïçÔß횊 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.rbd0.out‚ 3.087890625 disk.rbd2.out‚ 3.025390625! disk.nvme1n1.out‚ 70042.640625 network.recv‚ 3.949004e+06 disk.dm-0.in‚531405.14453125 disk.md0p1.in‚ 10.97265625 disk.sdb2.in‚5.40625% memory_percent‚39.290714263916016 disk.sdb1.in‚0 disk.sdb.in‚5.40625" proc.memory.rssMB‚ 3031.2421875 disk.sda.in‚ 5.56640625# disk.nvme0n1.out‚67056.40234375! disk.dm-0.out‚137099.04296875 disk.md0.in‚ 10.97265625) disk./.usagePercent‚83.18480498700276 disk.sdb1.out‚0# disk.nvme0n1.in‚329983.51171875 disk.md0.out‚ 1211.90234375 disk.sdb.out‚1257.6005859375! disk.sda2.out‚1257.6005859375 disk.md0p1.out‚ 1211.90234375 network.sent‚ 2.5622682e+07 disk.sda1.out‚0! disk.sdb2.out‚1257.6005859375' proc.memory.availableMB‚ 9946.609375) proc.memory.percent‚18.50123405456543 cpu‚25.199272316877142 proc.cpu.threads‚17" disk.nvme1n1.in‚201421.6328125$ disk./.usageGB‚2975.091552734375 disk.sda.out‚1257.6005859375 disk.sda1.in‚0 disk.sda2.in‚ 5.56640625”× ó4´(/ _wandbruntime‚11437  _runtime‚11437ƒÈüíX*µ(ÒT OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚379 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚19556121222656 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚19232294007808 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚19232118195712 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 347418624 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚582 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚394763750007296 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 7134511104 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6966900736 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚582 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 447555175 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 447534462 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6561988608 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 263210718 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 180222418 ZLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚ 110220644 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6741059457 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884753840994356e+09 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 193815040 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚19232294007808 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 180222039 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 560288777 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193815040 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚413995868203008 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3532 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 184323744 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚437659689117184 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 184323746 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20711 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 142955008 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚309 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚70001395 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚309 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32006 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 447534462 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚19177766583294 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3532 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚394763768010240 ^PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚ 110220646 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚375568981663232 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20711 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 263210718 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188859832 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚70001772 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 572522496 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚413996062018048 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6966900736 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚394763750007296 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 447555175 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚36 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚377 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 175812096 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚394763768010240 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 166895104~H]*¶(×T OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚379 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚19556121222656 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚19232294007808 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚19232118195712 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 347418624 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚582 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚394763750007296 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 7134511104 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6966900736 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚582 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 447555175 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 447534462 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6561988608 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 263210718 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 180222418 ZLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚ 110220644 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6741059457 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884753840994356e+09 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 193815040 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚19232294007808 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 180222039 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 560288777 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193815040 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚413995868203008 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3532 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 184323744 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚437659689117184 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 184323746 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20711 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 142955008 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚309 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚70001395 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚309 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32006 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 447534462 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚19177766583294 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3532 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚394763768010240 ^PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚ 110220646 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚375568981663232 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20711 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 263210718 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188859832 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚70001772 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 572522496 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚413996062018048 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6966900736 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚394763750007296 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 447555175 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚36 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚377 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 175812096 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚394763768010240 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 166895104™ßW“ª4·(/ _wandbruntime‚11438  _runtime‚11438yÇüݸ(ÚR cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚19556595082752 trainer/global_step‚9671 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 7134511104 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚394793833684480 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 572522496 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3532 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 263219626 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚414026601552384 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚582 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚19556738037760 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6741059457 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 263240337 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20713 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 447558752 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚582 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚273 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1319 \NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚ 110232328 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188859832 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚394772591244138 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32006 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6966900736 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚375594524273504 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚309 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚414026407737344 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 565025792 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 263219626 _step‚794 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 193815040 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 142955008 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚394772780103970 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚309 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚418133786693120 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚309 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚19178238791106 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚19232592055808 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚19232767867904 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚273 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚380 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 180239463 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚437690357835776 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 171820472 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚273 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 565025792 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚378 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 166895104 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚418133762753024 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚414026601552384 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚414026407737344 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20711 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 175812096 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 447579465 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 180239843 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 263240337 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 347418624 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _runtime‚11438.332434895 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚375594541312864 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 447579465 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚70007515 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 7134511104 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6561988608 # _timestamp‚1.78847538514648e+09 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 184339128 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚36˜áÓe)¹(ßR RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚582 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚19556738037760 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6741059457 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 263240337 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20713 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 447558752 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚582 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚273 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1319 \NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚ 110232328 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188859832 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚394772591244138 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32006 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6966900736 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚375594524273504 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚309 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚414026407737344 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 565025792 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 263219626 _step‚794 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 193815040 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 142955008 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚394772780103970 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚309 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚418133786693120 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚309 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚19178238791106 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚19232592055808 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚19232767867904 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚273 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚380 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 180239463 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚437690357835776 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 171820472 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚273 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 565025792 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚378 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 166895104 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚418133762753024 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚414026601552384 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚414026407737344 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20711 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 175812096 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 447579465 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 180239843 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 263240337 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 347418624 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _runtime‚11438.332434895 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚375594541312864 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 447579465 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚70007515 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 7134511104 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6561988608 # _timestamp‚1.78847538514648e+09 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 184339128 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚36 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚19556595082752 trainer/global_step‚9671 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 7134511104 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚394793833684480 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 572522496 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3532 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 263219626 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚414026601552384šZ椒º(‚(:‡ ÿïçÔ›žÐˆ gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚105.443' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚26.36075 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n>K €»(‚(:õ ÿïçÔÖÑ“‰ disk.md0.out‚ 1213.44140625 disk.sdb.out‚1259.193359375 disk.sdb1.in‚0 disk.sda.in‚ 5.56640625" proc.memory.rssMB‚ 3031.2421875 network.sent‚ 2.5638513e+07 disk.sdb2.out‚1259.193359375 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 3.025390625 disk.sdb.in‚5.40625 disk.sda.out‚1259.193359375 cpu‚25.204135100781937 disk.sdb1.out‚0 disk.sda2.in‚ 5.56640625 proc.cpu.threads‚17 disk.rbd0.out‚ 3.087890625# disk.nvme0n1.in‚329983.51171875! disk.nvme0n1.out‚ 67128.453125) proc.memory.percent‚18.50123405456543$ disk./.usageGB‚2975.093475341797 disk.md0.in‚ 10.97265625" disk.nvme1n1.out‚ 70113.5078125( proc.memory.availableMB‚ 9946.4765625 network.recv‚ 3.952366e+06 disk.rbd2.in‚0 disk.dm-0.in‚531405.14453125) disk./.usagePercent‚83.18485874391097 disk.dm-0.out‚137241.9609375 disk.md0p1.in‚ 10.97265625 disk.md0p1.out‚ 1213.44140625" disk.nvme1n1.in‚201421.6328125 disk.sda1.in‚0 disk.sda2.out‚1259.193359375$ memory_percent‚39.29152488708496 disk.sda1.out‚0 disk.sdb2.in‚5.40625zõÒ4¼(/ _wandbruntime‚11454  _runtime‚11454•óB¨A½(¸¦ ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3953703045845032 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.18769189715385437 +jet/accuracy_3_of_4‚0.08152119070291519 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ,particle/sensitivity‚0.34665125608444214 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9888848066329956 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0011337868636474013 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_6‚0.9807692170143127 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10087718814611435 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.992834210395813 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962748885154724 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5423646569252014 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.007998527027666569 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5632561445236206 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5739641785621643 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6559383273124695 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5810689330101013 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2093026489019394 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5468926429748535 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9874281287193298 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2842462956905365 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6121219992637634 particle/accuracy_3_of_5‚1 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.30747124552726746 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 _step‚795 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9876366853713989 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01758832484483719 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.012152778916060925 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9918614625930786 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5272586345672607 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.42546823620796204 *jet/accuracy_1_of_1‚0.5025332570075989 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_7‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0833333358168602 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.23437832295894623 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1379491537809372 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4967287480831146 *jet/accuracy_3_of_5‚0.1318100243806839 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0833333358168602 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4660660922527313 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 'particle/accuracy‚0.856321394443512 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 epoch‚7 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_3‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24554796516895294 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.48153939843177795 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.17176014184951782 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5880751013755798 8"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.736842155456543 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5387722253799438 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998443067073822 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9885613322257996 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.10991337895393372 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13896377384662628 particle/accuracy_1_of_2‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 jet/accuracy_5_of_6‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008789216168224812 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.19696968793869019 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3075202703475952 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4927960932254791 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6350127458572388 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 )loss/val_total_loss‚40.82111681618329 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5542250871658325 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_6_of_7‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.3094824552536011 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.05844597890973091 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.04402545839548111 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.997452974319458 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.111111119389534 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.16235633194446564 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7092283964157104 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9841525554656982 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1768391728401184 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.026242438703775406 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24056406319141388 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03673195093870163 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0011337868636474013 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.5097466111183167 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 +particle/specificity‚0.9506264328956604 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2499999701976776 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.20471730828285217 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14110292494297028 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6984126567840576 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.12547892332077026 particle/accuracy_1_of_1‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.8571428656578064 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7121211886405945 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_7‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.20366057753562927 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.010758377611637115 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5520833730697632 jet/accuracy_4_of_6‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980872869491577 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.11750134825706482 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt2SRqqt2FBt/event_purity‚0.0714285746216774 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 6 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.156849205493927 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.03202451765537262 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.041901521384716034 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5314039587974548 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.992976725101471 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5471727848052979 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2851029336452484 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.14848442375659943 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5701668858528137 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21517467498779297 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5208333134651184 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5299999713897705 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5857062339782715 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9694443941116333 particle/accuracy_2_of_5‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_6‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.15909092128276825 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 particle/accuracy_4_of_6‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.21623927354812622 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.024092968553304672 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ,"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.375 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23414163291454315 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.4000000059604645 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.32357925176620483 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.4461618661880493 particle/accuracy_5_of_6‚1 jet/accuracy_6_of_7‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5138843655586243 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09281455725431442 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1386055052280426 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ,"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9747983813285828 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1111111119389¼ƒÀñ—Q534 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.46492263674736023 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.5097701549530029 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2410714328289032 *jet/accuracy_2_of_5‚0.5256912708282471 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5444227457046509 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_2‚0.12190744280815125 *jet/accuracy_2_of_3‚0.2763291597366333 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3703703284263611 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5222077369689941 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16455549001693726 *jet/accuracy_1_of_4‚0.9319798946380615 ,jet/accuracy_4_of_5‚0.014426522888243198 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3430746793746948 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.42425599694252014 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0833333358168602 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2013927847146988 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5947249531745911 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17525292932987213 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 particle/accuracy_1_of_5‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 trainer/global_step‚9671 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3708929121494293 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5523248314857483 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.822398841381073 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6009783744812012 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.30101248621940613 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4970930218696594 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5661190748214722 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2173302322626114 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.19481980800628662 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.20366057753562927 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9770114421844482 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4469697177410126 *"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5996527671813965 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6383928060531616 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_7‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3312124013900757 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5480967164039612 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9883618950843811 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.030479973182082176 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5556604862213135 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.10043132305145264 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 *jet/accuracy_1_of_3‚0.8873125910758972 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.014367816969752312 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 _runtime‚11454.86599942 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9916207790374756 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4721134901046753 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.020863397046923637 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.5124020576477051 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5382313132286072 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.47657954692840576 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15657690167427063 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20836228132247925 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.518497109413147 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9918607473373413 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5277646780014038 particle/accuracy_1_of_4‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9903996586799622 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.05696316808462143 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9906809329986572 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.39405569434165955 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 $ _timestamp‚1.788475401674579e+09 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 (particle/f_score‚0.42954394221305847 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_2_of_7‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.4166666865348816 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.14891040325164795 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5860287547111511 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6167837381362915 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14785683155059814 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.09375 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05773809552192688 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9078947305679321 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.31111112236976624 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 particle/accuracy_7_of_7‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004682611208409071 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43827158212661743 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.46666663885116577 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17273813486099243 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9937641620635986 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5679011940956116 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9322916865348816 particle/accuracy_3_of_6‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5315436124801636 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5702508687973022 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9914259314537048 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0038197096437215805 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 jet/accuracy_5_of_5‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1354166567325592 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333730697632 particle/accuracy_2_of_6‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5714285969734192 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5523697137832642 8"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.603205144405365 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.6034482717514038 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5525712966918945 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0015873016091063619 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.46666666865348816 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9219858646392822 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1651315540075302 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4970237612724304 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 jet/accuracy_3_of_7‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4295731782913208 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_3‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1111111119389534 particle/accuracy_6_of_6‚1 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.057060759514570236 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5436781644821167 particle/accuracy_3_of_3‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 particle/accuracy_5_of_7‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.645957887172699 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.23187364637851715 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5435188412666321 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.392098069190979 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5940697193145752 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 *jet/accuracy_2_of_6‚0.6987179517745972 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.14678363502025604 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9522913694381714 :"Purity/*FRt2SRqqt2FBt/event_purity‚0.03999999910593033 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1644444465637207 particle/accuracy_1_of_6‚1 particle/accuracy_4_of_7‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17249836027622223 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_4_of_7‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.30320778489112854 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.18881218135356903 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5437597632408142 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16487468779087067 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.47626256942749023 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.11008229851722717 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08865249156951904 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4252645671367645 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5405701994895935 )jet/accuracy_1_of_2‚0.768938422203064 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_2_of_7‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 8"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.577329695224762 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 7validation_average_jet_accuracy‚0.45318493247032166 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.990686297416687 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11293560266494751 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.43033257126808167 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5277646780014038 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11657807230949402 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7424241900444031 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.40014880895614624 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.27876362204551697 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.27500003576278687 jet/accuracy_1_of_7‚0.5 jet/accuracy_7_of_7‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.20387740433216095 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.26530081033706665 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5538138151168823 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4076879322528839 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3910290598869324 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 +jet/accuracy_3_of_6‚0.10897436738014221 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5364631414413452 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9894494414329529 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.39548611640930176 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4789760410785675 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5365816950798035}ZHç[.¾(½¦ :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.47657954692840576 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15657690167427063 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20836228132247925 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.518497109413147 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9918607473373413 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5277646780014038 particle/accuracy_1_of_4‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9903996586799622 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.05696316808462143 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9906809329986572 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.39405569434165955 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 $ _timestamp‚1.788475401674579e+09 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 (particle/f_score‚0.42954394221305847 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_2_of_7‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.4166666865348816 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.14891040325164795 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5860287547111511 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6167837381362915 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14785683155059814 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.09375 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05773809552192688 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9078947305679321 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.31111112236976624 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 particle/accuracy_7_of_7‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004682611208409071 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43827158212661743 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.46666663885116577 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17273813486099243 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9937641620635986 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5679011940956116 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9322916865348816 particle/accuracy_3_of_6‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5315436124801636 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5702508687973022 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9914259314537048 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0038197096437215805 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 jet/accuracy_5_of_5‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1354166567325592 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333730697632 particle/accuracy_2_of_6‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5714285969734192 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5523697137832642 8"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.603205144405365 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.6034482717514038 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5525712966918945 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0015873016091063619 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.46666666865348816 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9219858646392822 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1651315540075302 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4970237612724304 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 jet/accuracy_3_of_7‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4295731782913208 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_3‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1111111119389534 particle/accuracy_6_of_6‚1 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.057060759514570236 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5436781644821167 particle/accuracy_3_of_3‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 particle/accuracy_5_of_7‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.645957887172699 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.23187364637851715 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5435188412666321 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.392098069190979 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5940697193145752 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 *jet/accuracy_2_of_6‚0.6987179517745972 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.14678363502025604 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9522913694381714 :"Purity/*FRt2SRqqt2FBt/event_purity‚0.03999999910593033 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1644444465637207 particle/accuracy_1_of_6‚1 particle/accuracy_4_of_7‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17249836027622223 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_4_of_7‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.30320778489112854 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.18881218135356903 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5437597632408142 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16487468779087067 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.47626256942749023 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.11008229851722717 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08865249156951904 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4252645671367645 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5405701994895935 )jet/accuracy_1_of_2‚0.768938422203064 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_2_of_7‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 8"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.577329695224762 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 7validation_average_jet_accuracy‚0.45318493247032166 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.990686297416687 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11293560266494751 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.43033257126808167 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5277646780014038 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11657807230949402 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7424241900444031 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.40014880895614624 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.27876362204551697 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.27500003576278687 jet/accuracy_1_of_7‚0.5 jet/accuracy_7_of_7‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.20387740433216095 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.26530081033706665 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5538138151168823 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4076879322528839 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3910290598869324 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 +jet/accuracy_3_of_6‚0.10897436738014221 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0zŽÂéd.5364631414413452 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9894494414329529 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.39548611640930176 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4789760410785675 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5365816950798035 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3953703045845032 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.18769189715385437 +jet/accuracy_3_of_4‚0.08152119070291519 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ,particle/sensitivity‚0.34665125608444214 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9888848066329956 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0011337868636474013 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_6‚0.9807692170143127 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10087718814611435 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.992834210395813 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962748885154724 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5423646569252014 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.007998527027666569 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5632561445236206 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5739641785621643 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6559383273124695 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5810689330101013 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2093026489019394 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5468926429748535 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9874281287193298 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2842462956905365 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6121219992637634 particle/accuracy_3_of_5‚1 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.30747124552726746 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 _step‚795 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9876366853713989 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01758832484483719 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.012152778916060925 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9918614625930786 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5272586345672607 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.42546823620796204 *jet/accuracy_1_of_1‚0.5025332570075989 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_7‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0833333358168602 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.23437832295894623 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1379491537809372 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4967287480831146 *jet/accuracy_3_of_5‚0.1318100243806839 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0833333358168602 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4660660922527313 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 'particle/accuracy‚0.856321394443512 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 epoch‚7 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_3‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24554796516895294 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.48153939843177795 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.17176014184951782 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5880751013755798 8"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.736842155456543 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5387722253799438 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998443067073822 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9885613322257996 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.10991337895393372 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13896377384662628 particle/accuracy_1_of_2‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 jet/accuracy_5_of_6‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008789216168224812 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.19696968793869019 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3075202703475952 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4927960932254791 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6350127458572388 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 )loss/val_total_loss‚40.82111681618329 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5542250871658325 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_6_of_7‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.3094824552536011 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.05844597890973091 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.04402545839548111 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.997452974319458 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.111111119389534 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.16235633194446564 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7092283964157104 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9841525554656982 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1768391728401184 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.026242438703775406 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24056406319141388 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03673195093870163 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0011337868636474013 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.5097466111183167 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 +particle/specificity‚0.9506264328956604 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2499999701976776 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.20471730828285217 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14110292494297028 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6984126567840576 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.12547892332077026 particle/accuracy_1_of_1‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.8571428656578064 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7121211886405945 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_7‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.20366057753562927 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.010758377611637115 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5520833730697632 jet/accuracy_4_of_6‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980872869491577 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.11750134825706482 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt2SRqqt2FBt/event_purity‚0.0714285746216774 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 6 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.156849205493927 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.03202451765537262 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.041901521384716034 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5314039587974548 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.992976725101471 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5471727848052979 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2851029336452484 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.14848442375659943 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5701668858528137 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21517467498779297 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5208333134651184 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5299999713897705 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5857062339782715 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9694443941116333 particle/accuracy_2_of_5‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_6‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.15909092128276825 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 particle/accuracy_4_of_6‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.21623927354812622 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.024092968553304672 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ,"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.375 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23414163291454315 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.4000000059604645 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.32357925176620483 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.4461618661880493 particle/accuracy_5_of_6‚1 jet/accuracy_6_of_7‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5138843655586243 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09281455725431442 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1386055052280426 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ,"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9747983813285828 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1111111119389534 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.46492263674736023 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.5097701549530029 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2410714328289032 *jet/accuracy_2_of_5‚0.5256912708282471 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5444227457046509 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_2‚0.12190744280815125 *jet/accuracy_2_of_3‚0.2763291597366333 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3703703284263611 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5222077369689941 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16455549001693726 *jet/accuracy_1_of_4‚0.9319798946380615 ,jet/accuracy_4_of_5‚0.014426522888243198 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3430746793746948 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.42425599694252014 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0833333358168602 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2013927847146988 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5947249531745911 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17525292932987213 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 particle/accuracy_1_of_5‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 trainer/global_step‚9671 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3708929121494293 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5523248314857483 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.822398841381073 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6009783744812012 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.30101248621940613 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4970930218696594 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5661190748214722 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2173302322626114 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.19481980800628662 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.20366057753562927 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9770114421844482 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4469697177410126 *"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5996527671813965 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6383928060531616 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_7‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3312124013900757 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5480967164039612 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9883618950843811 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.030479973182082176 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5556604862213135 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.10043132305145264 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 *jet/accuracy_1_of_3‚0.8873125910758972 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.014367816969752312 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 _runtime‚11454.86599942 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9916207790374756 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4721134901046753 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.020863397046923637 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.5124020576477051 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5382313132286072 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0›ì9‘¿(‚(:† ŽðçÔ·¨òˆ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚98.591' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚24.64775 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡R|…À(‚(:ú ŽðçÔíìÛ‰# disk.nvme0n1.in‚329983.51171875 disk.sda2.out‚1261.193359375 disk.md0p1.in‚ 10.97265625 disk.rbd2.out‚ 3.025390625 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 2.5733776e+07 disk.md0.out‚ 1215.36328125 disk.rbd0.in‚0.015625$ memory_percent‚39.29946422576904) proc.memory.availableMB‚ 9945.17578125) disk./.usagePercent‚83.18487751616462 disk.sdb.in‚5.40625 disk.sda.out‚1261.193359375 disk.dm-0.in‚531405.14453125 network.recv‚ 3.960294e+06# disk.nvme1n1.out‚70422.30078125 cpu‚25.207433381343243" disk.nvme1n1.in‚201421.6328125 disk.md0p1.out‚ 1215.36328125) proc.memory.percent‚18.50724220275879% disk./.usageGB‚2975.0941467285156 disk.sdb2.out‚1261.193359375 disk.sdb1.out‚0! disk.dm-0.out‚137623.26171875 disk.sda2.in‚ 5.56640625" proc.memory.rssMB‚ 3032.2265625 disk.md0.in‚ 10.97265625 disk.rbd0.out‚ 3.087890625 disk.rbd2.in‚0 disk.sdb.out‚1261.193359375 disk.sda.in‚ 5.56640625" disk.nvme0n1.out‚ 67200.9609375 proc.cpu.threads‚17 disk.sdb2.in‚5.40625 disk.sdb1.in‚0Lˆà‘Á(‚(:† ðçÔ¡ªÏ‡ gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚100.482' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.1205 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cǧ‡Â(‚(:ü ðçÔ‘›†ˆ" disk.nvme1n1.in‚201421.6328125 disk.rbd0.out‚ 3.099609375 network.sent‚ 2.5859176e+07 disk.sda1.in‚0) proc.memory.availableMB‚ 9944.97265625! disk.sda.out‚1263.04248046875 disk.rbd2.in‚0 disk.rbd2.out‚ 3.099609375 cpu‚25.207214083568378# disk.nvme0n1.in‚329983.51171875$ disk./.usageGB‚2974.816879272461 disk.sdb.in‚5.40625 disk.sda1.out‚0 disk.sda2.in‚ 5.56640625 disk.dm-0.out‚137734.0546875 disk.md0p1.out‚ 1217.1484375 disk.md0.out‚ 1217.1484375$ memory_percent‚39.30070400238037" proc.memory.rssMB‚ 3032.2265625 proc.cpu.threads‚17 disk.md0.in‚ 10.97265625 disk.rbd0.in‚0.015625" disk.nvme0n1.out‚ 67273.0078125 disk.sdb1.in‚0 disk.dm-0.in‚531405.14453125" disk.sda2.out‚1263.04248046875 disk.sdb2.in‚5.40625" disk.sdb2.out‚1263.04248046875 network.recv‚ 3.967943e+06 disk.sda.in‚ 5.56640625 disk.md0p1.in‚ 10.97265625) proc.memory.percent‚18.50724220275879) disk./.usagePercent‚83.17712500204789! disk.nvme1n1.out‚ 70461.046875! disk.sdb.out‚1263.04248046875 disk.sdb1.out‚0¬­a’Ã(‚(:‡ ¬ðçÔ¥¶º‡ gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.357' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.83925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷k±†Ä(‚(:û ¬ðçÔáóë‡ disk.dm-0.out‚ 137953.984375 network.recv‚ 3.972207e+06 disk.sda1.in‚0 disk.md0.in‚ 10.97265625" disk.nvme1n1.in‚201421.6328125# disk.nvme1n1.out‚70469.88671875! disk.sda.out‚1264.34033203125$ memory_percent‚39.30015563964844 proc.cpu.threads‚17$ disk./.usageGB‚2974.841651916504 disk.dm-0.in‚531405.14453125" disk.sda2.out‚1264.34033203125 disk.rbd2.in‚0 disk.sdb.in‚5.40625! disk.sdb.out‚1264.34033203125 disk.sdb1.in‚0% proc.memory.availableMB‚ 9945.0625 network.sent‚ 2.5865204e+07 disk.sda1.out‚0 disk.sdb2.in‚5.40625" disk.sdb2.out‚1264.34033203125# disk.nvme0n1.in‚329983.51171875# disk.nvme0n1.out‚67484.09765625) disk./.usagePercent‚83.17781765554348 disk.md0p1.in‚ 10.97265625 disk.md0.out‚ 1218.39453125" proc.memory.rssMB‚ 3032.2265625 disk.sda2.in‚ 5.56640625 disk.rbd0.in‚0.015625 cpu‚25.20696698123961 disk.md0p1.out‚ 1218.39453125 disk.rbd0.out‚ 3.162109375 disk.rbd2.out‚ 3.099609375 disk.sdb1.out‚0 disk.sda.in‚ 5.56640625) proc.memory.percent‚18.50724220275879FULÀ4Å(/ _wandbruntime‚11492  _runtime‚11492Ð(rÈÆ( % _timestamp‚1.7884754392397895e+09 _runtime‚11492.424862466 _step‚796 & lr-AdamW/pg1‚9.438665270397965e-05 & lr-AdamW/pg2‚9.438665270397965e-05 trainer/global_step‚9699NøF›ÍÇ(Ç _step‚796 & lr-AdamW/pg1‚9.438665270397965e-05 & lr-AdamW/pg2‚9.438665270397965e-05 trainer/global_step‚9699 % _timestamp‚1.7884754392397895e+09 _runtime‚11492.424862466œÆç)s4È(/ _wandbruntime‚11492  _runtime‚11492¨U¡€É(ëO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚396245410448384 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚697 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 185032519 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 185032517 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚419653932741120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚396245392445440 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 264081976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚396177404258322 % _timestamp‚1.78847543924117457º7q!e+09 _step‚797 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚70264848 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19291736128000 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 264081976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚70265543 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚396245392445440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚376940136408528 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 185032517 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚439270664113664 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚415537128573440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19291969796608 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19617019894784 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 110675846 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚415537128573440 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 180940694 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19237267849794 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _runtime‚11492.426886764 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19616731372544 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 264055632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚695 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚439270976576000 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚9699 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚415537380244992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 264055632 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19237038868710 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19291969796608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚419653956681216 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 264081976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚396177404258322 % _timestamp‚1.7884754392411745e+09 _step‚797 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚70264848 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19291736128000 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 264081976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚70265543 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚396245392445440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚376940136408528 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 185032517 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚439270664113664 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚415537128573440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19291969796608 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19617019894784 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 110675846 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚415537128573440 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 180940694 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19237267849794 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _runtime‚11492.426886764 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19616731372544 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 264055632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚695 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚439270976576000 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚9699 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚415537380244992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 264055632 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19237038868710 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19291969796608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚419653956681216 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 110686650 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚415578630321152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19239226338988 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚396284696485376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 449133116 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 185050792 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19619016160256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 449159488 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 264082324 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19293700153856 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚11493.612874699 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚396216647084288 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 185050792 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 264082324 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 449133116 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 180958679 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚419695640651264 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70272029 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884754404261665e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚415578630321152 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚396284696485376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19293700153856 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 110686648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 180958004 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚396284678482432 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚396284678482432 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 449159488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19618723457024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚439314340168192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚396216893104940 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19238997357696 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚415578378636288 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚439314656811520 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _step‚798 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19239226338988 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚396284696485376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 449133116 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 185050792 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19619016160256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 449159488 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 264082324 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19293700153856 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚11493.612874699 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚396216647084288 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 185050792 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 264082324 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 449133116 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 180958679 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚419695640651264 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70272029 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884754404261665e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚415578630321152 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚396284696485376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19293700153856 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 110686648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 180958004 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚396284678482432 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚396284678482432 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 449159488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19618723457024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚439314340168192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚396216893104940 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19238997357696 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚415578378636288 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚439314656811520 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _step‚798 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 110686650žgwáJ4Î(/ _wandbruntime‚11493  _runtime‚11493 üi_Ï(Ù 0loss/FBt1/assignment_loss‚0.2658660294779958 /loss/FBt3/detection_loss‚0.5288330786587561 epoch‚8 2loss/SRqqt2/assignment_loss‚1.5127151089495625 /loss/FRt2/detection_loss‚2.0622634037293266 0loss/SRqqt2/detection_loss‚2.342199502320125 _step‚799 0loss/FRt2/assignment_loss‚3.2017053158915774 2loss/SRqqt1/assignment_loss‚1.2401459745474552 1loss/SRqqt3/detection_loss‚0.8323930384746442 0loss/FRt3/assignment_loss‚1.1429602562541692 2loss/FBt3/assignment_loss‚0.061587056362930236 .loss/FRt1/detection_loss‚2.930090485938958 /loss/FBt4/detection_loss‚2.0368989725907642 % _timestamp‚1.7884754404293182e+09 2loss/SRqqt3/assignment_loss‚1.0645888739963993 1loss/SRqqt1/detection_loss‚2.0421088188886642 trainer/global_step‚9699 /loss/FRt1/assignment_loss‚2.777862470064844 2loss/SRqqt4/assignment_loss‚1.1531589209917001 0loss/FBt2/assignment_loss‚0.1469634783728124 1loss/SRqqt4/detection_loss‚2.7702559722321376 %loss/total_loss‚40.60548088492395 0loss/FRt4/assignment_loss‚2.7147478527492948 1loss/FBt4/assignment_loss‚0.24978797973599237 /loss/FRt3/detection_loss‚1.4442498643266064 /loss/FRt4/detection_loss‚2.8092352739638753 /loss/FBt2/detection_loss‚2.7764924331144853 _runtime‚11493.614220955 .loss/FBt1/detection_loss‚2.498370723290877’ÁþdÐ(Þ epoch‚8 2loss/SRqqt2/assignment_loss‚1.5127151089495625 /loss/FRt2/detection_loss‚2.0622634037293266 0loss/SRqqt2/detection_loss‚2.342199502320125 _step‚799 0loss/FRt2/assignment_loss‚3.2017053158915774 2loss/SRqqt1/assignment_loss‚1.2401459745474552 1loss/SRqqt3/detection_loss‚0.8323930384746442 0loss/FRt3/assignment_loss‚1.1429602562541692 2loss/FBt3/assignment_loss‚0.061587056362930236 .loss/FRt1/detection_loss‚2.930090485938958 /loss/FBt4/detection_loss‚2.0368989725907642 % _timestamp‚1.7884754404293182e+09 2loss/SRqqt3/assignment_loss‚1.0645888739963993 1loss/SRqqt1/detection_loss‚2.0421088188886642 trainer/global_step‚9699 /loss/FRt1/assignment_loss‚2.777862470064844 2loss/SRqqt4/assignment_loss‚1.1531589209917001 0loss/FBt2/assignment_loss‚0.1469634783728124 1loss/SRqqt4/detection_loss‚2.7702559722321376 %loss/total_loss‚40.60548088492395 0loss/FRt4/assignment_loss‚2.7147478527492948 1loss/FBt4/assignment_loss‚0.24978797973599237 /loss/FRt3/detection_loss‚1.4442498643266064 /loss/FRt4/detection_loss‚2.8092352739638753 /loss/FBt2/detection_loss‚2.7764924331144853 _runtime‚11493.614220955 .loss/FBt1/detection_loss‚2.498370723290877 0loss/FBt1/assignment_loss‚0.2658660294779958 /loss/FBt3/detection_loss‚0.5288330786587561ŸVÑ(‚(:‘ »ðçÔåáù† gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚118.474' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.618500000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êäæŒÒ(‚(:  »ðçÔÄŸµ‰" disk.sda2.out‚1266.37255859375" disk.sdb2.out‚1266.37255859375 disk.rbd0.out‚ 3.162109375 disk.sda1.in‚0$ memory_percent‚39.30180072784424) proc.memory.percent‚18.50724220275879 cpu‚25.206734086282573) disk./.usagePercent‚83.17962512492082 disk.md0p1.in‚ 10.97265625 disk.sdb2.in‚5.40625 disk.md0.out‚ 1220.3671875 network.sent‚ 2.5881078e+07 disk.md0.in‚ 10.97265625 disk.rbd0.in‚0.015625 disk.dm-0.out‚138190.7421875) proc.memory.availableMB‚ 9944.79296875 disk.sdb.in‚5.40625! disk.sdb.out‚1266.37255859375 disk.sdb1.in‚0" proc.memory.rssMB‚ 3032.2265625 proc.cpu.threads‚17 disk.md0p1.out‚ 1220.3671875# disk.nvme1n1.out‚70632.59765625 disk.sda.in‚ 5.56640625! disk.sda.out‚1266.37255859375# disk.nvme0n1.out‚67558.14453125 disk.sda1.out‚0 network.recv‚ 3.975839e+06 disk.sda2.in‚ 5.56640625 disk.rbd2.in‚0 disk.rbd2.out‚ 3.20166015625" disk.nvme1n1.in‚201421.6328125 disk.sdb1.out‚0# disk.nvme0n1.in‚329983.51171875 disk.dm-0.in‚531405.14453125$ disk./.usageGB‚2974.906295776367”µ’Ó(‚(:‡ ÊðçÔðч gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚105.443' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚26.36075 gpu.0.smClock‚1215 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò4&™†Ô(‚(:û ÊðçÔƒÖµˆ proc.cpu.threads‚17 disk.sda.in‚ 5.56640625 disk.md0p1.out‚ 1221.84375" proc.memory.rssMB‚ 3032.2265625 disk.rbd0.in‚0.015625 disk.sdb.in‚5.40625 disk.sdb1.in‚0 disk.md0p1.in‚ 10.97265625 disk.sda1.out‚0" disk.sda2.out‚1267.91357421875 network.recv‚ 3.983286e+06 disk.md0.out‚ 1221.84375" disk.nvme1n1.in‚201421.6328125# disk.nvme0n1.in‚329983.51171875 network.sent‚ 2.5978528e+07 disk.sdb2.in‚5.40625 disk.sda1.in‚0 disk.sda2.in‚ 5.56640625' proc.memory.availableMB‚ 9944.984375 disk.rbd2.out‚ 3.20166015625" disk.nvme1n1.out‚ 70632.9765625# disk.nvme0n1.out‚67716.33984375 disk.dm-0.in‚531405.14453125$ memory_percent‚39.30063247680664) proc.memory.percent‚18.50724220275879 cpu‚25.20651355868728! disk.dm-0.out‚138349.31640625( disk./.usagePercent‚83.1848602371584" disk.sdb2.out‚1267.91357421875 disk.md0.in‚ 10.97265625 disk.rbd2.in‚0! disk.sdb.out‚1267.91357421875% disk./.usageGB‚2975.0935287475586 disk.rbd0.out‚ 3.2255859375 disk.sdb1.out‚0! disk.sda.out‚1267.91357421875oº…‘Õ(‚(:† ÙðçÔøØŇ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚100.482' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.1205 gpu.0.smClock‚1170 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ú«]í{Ö(‚(:ð ÙðçÔœªˆ disk.sdb.in‚5.40625" proc.memory.rssMB‚ 3032.2265625 network.recv‚ 3.987742e+06 disk.sdb2.out‚ 1269.4765625" disk.nvme1n1.in‚201421.6328125 disk.sdb1.in‚0 disk.sda.out‚ 1269.4765625 cpu‚25.20626775232227 network.sent‚ 2.5984398e+07) disk./.usagePercent‚83.18485874391097 disk.rbd2.out‚ 3.2412109375) proc.memory.availableMB‚ 9944.55078125 proc.cpu.threads‚17" disk.nvme1n1.out‚ 70633.0859375 disk.sdb1.out‚0 disk.dm-0.in‚531405.14453125 disk.sda1.out‚0 disk.sda2.out‚ 1269.4765625$ disk./.usageGB‚2975.093475341797 disk.sda.in‚ 5.56640625# disk.nvme0n1.in‚329983.51171875! disk.dm-0.out‚138519.11328125 disk.sda2.in‚ 5.56640625 disk.md0p1.in‚ 10.97265625 disk.rbd0.out‚ 3.2412109375 disk.rbd2.in‚0 disk.sda1.in‚0 disk.md0p1.out‚ 1223.3515625 disk.sdb2.in‚5.40625 disk.rbd0.in‚0.015625 disk.sdb.out‚ 1269.4765625# disk.nvme0n1.out‚67886.02734375$ memory_percent‚39.30327892303467) proc.memory.percent‚18.50724220275879 disk.md0.in‚ 10.97265625 disk.md0.out‚ 1223.35156253‚ŽJ‘×(‚(:† èðçԤᘈ gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚32.0 gpu.0.powerWatts‚90.513' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚22.62825 gpu.0.smClock‚1200 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}šJN‹Ø(‚(:€  èðçÔƒŠßˆ disk.md0p1.in‚ 10.97265625 disk.sdb.in‚5.40625 disk.sdb1.in‚0# disk.nvme0n1.in‚329983.51171875! disk.dm-0.out‚138706.18359375 cpu‚25.206041943792112 proc.cpu.threads‚17$ disk./.usageGB‚2975.093475341797 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 3.30419921875" proc.memory.rssMB‚ 3032.2265625! disk.sdb.out‚1271.69970703125" disk.sda2.out‚1271.69970703125 disk.rbd2.in‚0 disk.dm-0.in‚531405.14453125 disk.md0.in‚ 10.97265625 disk.md0.out‚ 1225.5078125 disk.sda2.in‚ 5.56640625 disk.sdb2.in‚5.40625" disk.sdb2.out‚1271.69970703125) disk./.usagePercent‚83.18485874391097 disk.sdb1.out‚0 disk.sda.in‚ 5.56640625! disk.sda.out‚1271.69970703125) proc.memory.availableMB‚ 9945.04296875 network.sent‚ 2.599046e+07# disk.nvme1n1.out‚70633.26171875 disk.sda1.in‚0) proc.memory.percent‚18.50724220275879" disk.nvme1n1.in‚201421.6328125! disk.nvme0n1.out‚ 68072.921875 disk.sda1.out‚0 disk.md0p1.out‚ 1225.5078125$ memory_percent‚39.30027484893799 network.recv‚ 3.992174e+06 disk.rbd2.out‚ 3.2412109375IÜØ—4Ù(/ _wandbruntime‚11553  _runtime‚11553Òº'ÈÚ( & lr-AdamW/pg2‚9.063035478481895e-05 trainer/global_step‚9749 % _timestamp‚1.7884755007344024e+09 _runtime‚11553.919459986 _step‚800 & lr-AdamW/pg1‚9.063035478481895e-05ð±AÈÍÛ(Ç trainer/global_step‚9749 % _timestamp‚1.7884755007344024e+09 _runtime‚11553.919459986 _step‚800 & lr-AdamW/pg1‚9.063035478481895e-05 & lr-AdamW/pg2‚9.063035478481895e-05 íÿÃ4Ü(/ _wandbruntime‚11553  _runtime‚11553æóbþð'Ý(êO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚398209712297984 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 181831586 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚441454908639744 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚378816637240368 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 265393584 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 265419928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 185946267 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7884755007351847e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 451366197 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚684 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚417599897801216 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 265393584 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19335205890302 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚417599646129664 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 181830902 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 111215946 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 185946269 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚417599897801216 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 265419928 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19389951834624 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 451339851 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19389951834624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚70614956 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚398209694295040 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 111215948 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19716777396224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚801 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19390185503232 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚421738155183616 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚9749 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚421738131243520 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19717070112768 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608*ßB}Þ(ïO XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19389951834624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚70614956 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚398209694295040 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 111215948 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19716777396224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚801 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19390185503232 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚421738155183616 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚9749 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚421738131243520 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19717070112768 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚398209712297984 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 181831586 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚441454908639744 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚378816637240368 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 265393584 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 265419928 JDDeviceStatsMonitor.on_train|î—x _batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 185946267 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7884755007351847e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 451366197 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚684 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚417599897801216 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 265393584 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19335205890302 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚417599646129664 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 181830902 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 111215946 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 185946269 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚417599897801216 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 265419928 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19389951834624 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 451339851 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344¡ìN±4ß(/ _wandbruntime‚11554  _runtime‚11554õmøcû&à(õM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 451411190 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 185964544 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 265446646 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 185964542 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19391915860480 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚11554.943021025 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 451411190 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 111226750 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚421779839153664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚378854167597792 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚398248998334976 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 111226748 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚441498896424448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚441498588169728 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19392149542400 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚417641147877376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 451384818 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚421779815213568 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 185964542 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 265420276 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚398191349016648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚398248998334976 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚398191102995996 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 181848832 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19392149542400 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚398248980332032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19718772956160 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚378854184637152 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚417641147877376 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19337164379496 trainer/global_step‚9749 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 451411190 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 111226750 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚421779839153664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚378854167597792 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚398248998334976 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 111226748 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚441498896424448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚441498588169728 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19392149542400 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚417641147877376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 451384818 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚421779815213568 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 185964542 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 265420276 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚398191349016648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚398248998334976 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚398191102995996 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 181848832 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19392149542400 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚398248980332032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19718772956160 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚378854184637152 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚417641147877376 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19337164379496 trainer/global_step‚9749 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚11554.943021025 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249¢–\7<4â(/ _wandbruntime‚11554  _runtime‚11554ð‡[H[ã(Õ 0loss/SRqqt2/detection_loss‚2.276200649706093 /loss/FBt2/detection_loss‚2.7401442660225763 /loss/FRt1/assignment_loss‚2.223102679378108 0loss/FBt1/assignment_loss‚0.3856863957452726 2loss/SRqqt3/assignment_loss‚0.8308675626932344 .loss/FRt2/detection_loss‚2.040630603164318 trainer/global_step‚9749 0loss/FRt2/assignment_loss‚2.4793602538108828 2loss/FBt3/assignment_loss‚0.054045250134779226 1loss/SRqqt3/detection_loss‚0.8313125925236626 /loss/FBt3/detection_loss‚0.5472467807680368 _step‚803 2loss/SRqqt1/assignment_loss‚1.3782538327866711 1loss/SRqqt2/assignment_loss‚1.286534719974608 0loss/SRqqt4/detection_loss‚2.464264062047005 1loss/FBt4/assignment_loss‚0.11691772765646238 0loss/SRqqt1/detection_loss‚2.195546329021454 /loss/FBt4/detection_loss‚1.9521039830786842 epoch‚8 2loss/FBt2/assignment_loss‚0.005685768682772461 /loss/FRt1/detection_loss‚3.2983490730586804 /loss/FRt3/detection_loss‚1.4736763339855574 /loss/FBt1/detection_loss‚2.4358423054218292 $loss/total_loss‚39.3020013462344 _runtime‚11554.944351181 0loss/FRt3/assignment_loss‚1.2574627963559968 .loss/FRt4/detection_loss‚2.850032939910889 $ _timestamp‚1.788475501759523e+09 /loss/FRt4/assignment_loss‚2.907986350059509 2loss/SRqqt4/assignment_loss‚1.2707480902473132üŠxF`ä(Ú /loss/FBt4/detection_loss‚1.9521039830786842 epoch‚8 2loss/FBt2/assignment_loss‚0.005685768682772461 /loss/FRt1/detection_loss‚3.2983490730586804 /loss/FRt3/detection_loss‚1.4736763339855574 /loss/FBt1/detection_loss‚2.4358423054218292 $loss/total_loss‚39.3020013462344 _runtime‚11554.944351181 0loss/FRt3/assignment_loss‚1.2574627963559968 .loss/FRt4/detection_loss‚2.850032939910889 $ _timestamp‚1.788475501759523e+09 /loss/FRt4/assignment_loss‚2.907986350059509 2loss/SRqqt4/assignment_loss‚1.2707480902473132 0loss/SRqqt2/detection_loss‚2.276200649706093 /loss/FBt2/detection_loss‚2.7401442660225763 /loss/FRt1/assignment_loss‚2.223102679378108 0loss/FBt1/assignment_loss‚0.3856863957452726 2loss/SRqqt3/assignment_loss‚0.8308675626932344 .loss/FRt2/detection_loss‚2.040630603164318 trainer/global_step‚9749 0loss/FRt2/assignment_loss‚2.4793602538108828 2loss/FBt3/assignment_loss‚0.054045250134779226 1loss/SRqqt3/detection_loss‚0.8313125925236626 /loss/FBt3/detection_loss‚0.5472467807680368 _step‚803 2loss/SRqqt1/assignment_loss‚1.3782538327866711 1loss/SRqqt2/assignment_loss‚1.286534719974608 0loss/SRqqt4/detection_loss‚2.464264062047005 1loss/FBt4/assignment_loss‚0.11691772765646238 0loss/SRqqt1/detection_loss‚2.195546329021454£½›‘å(‚(:† ÷ðçÔ”Êð† gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚116.306' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.0765 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùÃh°…æ(‚(:ú ÷ðçÔ••‹" disk.nvme1n1.in‚201421.6328125 disk.sdb1.in‚0 disk.sda.out‚1273.2412109375 proc.cpu.threads‚17 disk.sdb.in‚5.40625 disk.sda1.in‚0! disk.sda2.out‚1273.2412109375 disk.sdb.out‚1273.2412109375 disk.sda.in‚ 5.56640625" proc.memory.rssMB‚ 3032.2265625 cpu‚25.205807622480854# disk.nvme0n1.in‚329983.51171875! disk.nvme0n1.out‚ 68343.671875! disk.sdb2.out‚1273.2412109375 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.sdb1.out‚0$ memory_percent‚39.30172920227051( proc.memory.availableMB‚ 9944.8046875 disk.dm-0.out‚138977.0390625 network.sent‚ 2.6006493e+07 network.recv‚ 3.995952e+06 disk.rbd2.out‚ 3.30419921875) disk./.usagePercent‚83.17710793636273 disk.sdb2.in‚5.40625 disk.dm-0.in‚531405.14453125 disk.sda1.out‚0) proc.memory.percent‚18.50724220275879 disk.md0.in‚ 10.97265625" disk.nvme1n1.out‚ 70633.3671875 disk.sda2.in‚ 5.56640625 disk.md0p1.in‚ 10.97265625 disk.md0p1.out‚ 1226.984375 disk.md0.out‚ 1226.984375 disk.rbd0.out‚ 3.30419921875% disk./.usageGB‚2974.8162689208984OÒE›ç(‚(: †ñçÔÇÿ‡ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a¬ ò†è(‚(:û †ñçÔ¦Ïóˆ network.sent‚ 2.6103957e+07 disk.sda1.out‚0" disk.sda2.out‚1274.69970703125 disk.rbd0.in‚0.015625 disk.sda.in‚ 5.56640625" proc.memory.rssMB‚ 3032.2265625 network.recv‚ 4.003181e+06 disk.md0.in‚ 10.97265625 disk.sdb.in‚5.40625# disk.nvme0n1.in‚329983.51171875 disk.sda2.in‚ 5.56640625$ memory_percent‚39.30230140686035 proc.cpu.threads‚17 disk.md0.out‚ 1228.39453125! disk.dm-0.out‚139049.85546875" disk.sdb2.out‚1274.69970703125 cpu‚25.20557359949149% disk./.usageGB‚2974.7765350341797" disk.nvme1n1.in‚201421.6328125 disk.sdb1.in‚0 disk.rbd0.out‚ 3.3671875! disk.sdb.out‚1274.69970703125 disk.dm-0.in‚531405.14453125 disk.md0p1.in‚ 10.97265625 disk.md0p1.out‚ 1228.39453125 disk.sdb2.in‚5.40625) proc.memory.percent‚18.50724220275879# disk.nvme1n1.out‚70633.60546875 disk.sda1.in‚0( proc.memory.availableMB‚ 9944.7109375) disk./.usagePercent‚83.17599696026016 disk.rbd2.in‚0 disk.rbd2.out‚ 3.30419921875 disk.sdb1.out‚0! disk.sda.out‚1274.69970703125 disk.nvme0n1.out‚68416.25žkƒ’é(‚(:‡ •ñçÔ÷ ú† gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚123.435' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.85875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸DÙê(‚(:ô •ñçÔÓú»‡ disk.md0p1.in‚ 10.97265625 disk.md0p1.out‚ 1229.859375 disk.dm-0.out‚ 139188.828125' proc.memory.availableMB‚ 9944.703125 network.sent‚ 2.6107497e+07 disk.md0.in‚ 10.97265625 disk.sdb1.in‚0 disk.dm-0.in‚531405.14453125) proc.memory.percent‚18.50724220275879 disk.rbd2.in‚0 disk.sdb1.out‚0 network.recv‚ 4.006975e+06 disk.md0.out‚ 1229.859375 disk.rbd2.out‚ 3.3671875 disk.sdb.in‚5.40625! disk.sda.out‚1276.22509765625" proc.memory.rssMB‚ 3032.2265625 disk.sdb2.in‚5.40625! disk.sdb.out‚1276.22509765625 disk.sda1.out‚0" disk.sdb2.out‚1276.22509765625 disk.rbd0.out‚ 3.3671875" disk.nvme1n1.in‚201421.6328125 disk.sda.in‚ 5.56640625$ memory_percent‚39.30234909057617 cpu‚25.205379338389953) disk./.usagePercent‚83.17787471892817 disk.rbd0.in‚0.015625" disk.nvme1n1.out‚ 70633.7109375# disk.nvme0n1.in‚329983.51171875" disk.nvme0n1.out‚ 68555.1171875 disk.sda1.in‚0 disk.sda2.in‚ 5.56640625" disk.sda2.out‚1276.22509765625$ disk./.usageGB‚2974.843692779541 proc.cpu.threads‚17×Zv…4ë(/ _wandbruntime‚11609  _runtime‚116099T˯Èì( & lr-AdamW/pg1‚8.694107698926476e-05 & lr-AdamW/pg2‚8.694107698926476e-05 trainer/global_step‚9799 % _timestamp‚1.7884755566341505e+09 _runtime‚11609.819040063 _step‚8049¶{ßÍí(Ç trainer/global_step‚9799 % _timestamp‚1.7884755566341505e+09 _runtime‚11609.819040063 _step‚804 & lr-AdamW/pg1‚8.694107698926476e-05 & lr-AdamW/pg2‚8.694107698926476e-05¤ŠkQÑ4î(/ _wandbruntime‚11609  _runtime‚11609öø¶6§ï(êO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚419662415357440 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚419662163685888 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 266731536 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚70960190 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19488401209856 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 182716236 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ø xäI#19816661559296 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318753792 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚419662415357440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚685 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚443638991305216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚419662163685888 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚443639310059008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 266731536 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19488167541248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚380693172150928 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 186860019 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19432914949726 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19488401209856 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 182716921 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚400126070061294 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚380693155111568 $ _timestamp‚1.788475556635165e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚70960873 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 111756048 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 453591553—LÎßõ'ð(ïO TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19488167541248 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 186860017 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚400174014147584 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 186860019 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚805 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚400173996144640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚683 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294813696 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19433143930810 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚400126070061294 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚380693155111568 $ _timestamp‚1.788475556635165e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚70960873 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 111756048 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 453591553 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚419662415357440 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚419662163685888 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 266731536 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚70960190 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19488401209856 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 182716236 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19816661559296 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318753792 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚419662415357440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚685 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚443638991305216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚419662163685888 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚443639310059008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 266731536 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19488167541248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚380693172150928 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 186860019 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19432914949726 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19488401209856 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 182716921 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 453636520 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚400213300184576 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚400213282181632 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19490131567104 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 453662892 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19490131567104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 111766848 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 266758228 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚687 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 266784598 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚380730685468992 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19434873438712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19818947288064 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚380730702508352 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚400165804928356 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 453662892 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19435102420004 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 111766850 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚11610.769391488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 266784598 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 182734495 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19818656681984 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚400213300184576 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 182733808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _step‚806 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 186878292 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚400165804928356 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 453662892 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19435102420004 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 111766850 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚11610.769391488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 266784598 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 182734495 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19818656681984 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚400213300184576 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 182733808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _step‚806 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 186878292 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 453636520 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚400213300184576 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚400213282181632 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19490131567104 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 453662892 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19490131567104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 111766848 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 266758228 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚687 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 266784598 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚380730685468992 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19434873438712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19818947288064 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚380730702508352¦aŽ î4ö(/ _wandbruntime‚11610  _runtime‚11610ÐuøØ^÷(Ø 2loss/SRqqt3/assignment_loss‚0.8860957629237605 1loss/SRqqt3/detection_loss‚0.7655638272283218 /loss/FBt3/detection_loss‚0.5413927003586044 trainer/global_step‚9799 /loss/FRt2/assignment_loss‚3.145454967021942 0loss/FRt3/assignment_loss‚1.0772058367729187 2loss/FBt1/assignment_loss‚0.041477942984568014 /loss/FRt2/detection_loss‚2.1632922140255686 0loss/SRqqt2/detection_loss‚2.861879194560258 1loss/SRqqt4/detection_loss‚2.5141440767508287 2loss/FBt2/assignment_loss‚0.037101195986264504 /loss/FRt3/detection_loss‚1.3927549819892913 %loss/total_loss‚41.67990512230188 _step‚807 2loss/SRqqt2/assignment_loss‚1.8379257794307626 /loss/FRt1/assignment_loss‚2.786313941984466 0loss/FBt4/assignment_loss‚0.9230823232156699 .loss/FRt1/detection_loss‚2.815060991229433 1loss/SRqqt1/detection_loss‚1.9633140272714875 /loss/FBt4/detection_loss‚2.5165320309725674 0loss/FRt4/assignment_loss‚2.9209307905505684 0loss/FBt3/assignment_loss‚0.0147607843488312 2loss/SRqqt4/assignment_loss‚1.9561022409739404 .loss/FRt4/detection_loss‚2.736001377596575 /loss/FBt2/detection_loss‚1.8773334324359894 epoch‚8 _runtime‚11610.770864845 2loss/SRqqt1/assignment_loss‚1.4558066532693126 .loss/FBt1/detection_loss‚2.450378048419952 % _timestamp‚1.7884755575861678e+09’Eh\cø(Ý .loss/FRt1/detection_loss‚2.815060991229433 1loss/SRqqt1/detection_loss‚1.9633140272714875 /loss/FBt4/detection_loss‚2.5165320309725674 0loss/FRt4/assignment_loss‚2.9209307905505684 0loss/FBt3/assignment_loss‚0.0147607843488312 2loss/SRqqt4/assignment_loss‚1.9561022409739404 .loss/FRt4/detection_loss‚2.736001377596575 /loss/FBt2/detection_loss‚1.8773334324359894 epoch‚8 _runtime‚11610.770864845 2loss/SRqqt1/assignment_loss‚1.4558066532693126 .loss/FBt1/detection_loss‚2.450378048419952 % _timestamp‚1.7884755575861678e+09 2loss/SRqqt3/assignment_loss‚0.8860957629237605 1loss/SRqqt3/detection_loss‚0.7655638272283218 /loss/FBt3/detection_loss‚0.5413927003586044 trainer/global_step‚9799 /loss/FRt2/assignment_loss‚3.145454967021942 0loss/FRt3/assignment_loss‚1.0772058367729187 2loss/FBt1/assignment_loss‚0.041477942984568014 /loss/FRt2/detection_loss‚2.1632922140255686 0loss/SRqqt2/detection_loss‚2.861879194560258 1loss/SRqqt4/detection_loss‚2.5141440767508287 2loss/FBt2/assignment_loss‚0.037101195986264504 /loss/FRt3/detection_loss‚1.3927549819892913 %loss/total_loss‚41.67990512230188 _step‚807 2loss/SRqqt2/assignment_loss‚1.8379257794307626 /loss/FRt1/assignment_loss‚2.786313941984466 0loss/FBt4/assignment_loss‚0.9230823232156699§êždù(‚(:… ³ñçÔçËý‡ gpu.0.gpu‚22.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚146.048' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.512 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 fª}ú(‚(:ò ³ñçÔù•¶ˆ disk.sda1.out‚0 disk.md0p1.in‚ 10.97265625 cpu‚25.20488942624481 proc.cpu.threads‚17" disk.nvme1n1.out‚ 70634.0703125 disk.sdb1.in‚0% memory_percent‚39.299726486206055) proc.memory.percent‚18.50724220275879 disk.dm-0.out‚139473.2734375 disk.sdb2.in‚5.40625! disk.sda.out‚1279.69677734375 network.sent‚ 2.622133e+07$ disk./.usageGB‚2974.031852722168 disk.rbd2.out‚ 3.3671875! disk.nvme0n1.out‚ 68839.203125 disk.dm-0.in‚531405.14453125 disk.md0p1.out‚ 1233.21875 disk.md0.out‚ 1233.21875 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 3.3671875) disk./.usagePercent‚83.15517533114254 disk.rbd2.in‚0# disk.nvme0n1.in‚329983.51171875( proc.memory.availableMB‚ 9945.1328125 disk.sdb1.out‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 3032.2265625" disk.sdb2.out‚1279.69677734375 disk.md0.in‚ 10.97265625 disk.sdb.in‚5.40625 disk.sda2.in‚ 5.56640625" disk.sda2.out‚1279.69677734375 network.recv‚ 4.017355e+06" disk.nvme1n1.in‚201421.6328125! disk.sdb.out‚1279.69677734375 disk.sda.in‚ 5.56640625i“z‘û(‚(:† ÂñçÔÓŽ¼‡ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚140.478' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.1195 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢§„’oü(‚(:ä ÂñçÔ¯Œò‡ disk.sdb.in‚5.40625 disk.sda1.out‚0( proc.memory.availableMB‚ 9944.8984375" proc.memory.rssMB‚ 3032.2265625 disk.rbd0.out‚ 3.3671875) disk./.usagePercent‚83.15517490450043! disk.nvme0n1.out‚ 68977.140625 disk.md0.in‚ 10.97265625) proc.memory.percent‚18.50724220275879$ disk./.usageGB‚2974.031837463379 disk.sdb1.out‚0# disk.nvme0n1.in‚329983.51171875 disk.md0p1.in‚ 10.97265625 disk.sdb2.out‚ 1281.76953125" disk.nvme1n1.in‚201421.6328125 disk.rbd2.out‚ 3.3671875 cpu‚25.204649957808364 disk.sdb.out‚ 1281.76953125 disk.dm-0.in‚531405.14453125 disk.rbd0.in‚0.015625 network.recv‚ 4.024582e+06 disk.sda.in‚ 5.56640625 disk.dm-0.out‚ 139611.390625 disk.sda1.in‚0 disk.sda2.in‚ 5.56640625 disk.rbd2.in‚0 network.sent‚ 2.6308246e+07 disk.md0p1.out‚ 1235.234375 disk.md0.out‚ 1235.234375% memory_percent‚39.301156997680664 disk.sdb1.in‚0 disk.sda.out‚ 1281.76953125 disk.sda2.out‚ 1281.76953125 disk.sdb2.in‚5.40625 proc.cpu.threads‚17 disk.nvme1n1.out‚70634.25½Ý‘ý(‚(:† ÑñçÔ´Ÿé… gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚134.298' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.5745 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àœÆˆþ(‚(:ý ÑñçÔ󹑇) proc.memory.percent‚18.50724220275879 disk.rbd2.out‚ 3.3671875" proc.memory.rssMB‚ 3032.2265625 cpu‚25.204412177469504# disk.nvme1n1.out‚70634.34765625 disk.sda2.in‚ 5.56640625' proc.memory.availableMB‚ 9944.640625! disk.sda.out‚1283.15966796875 disk.dm-0.out‚139650.2265625 disk.sda1.out‚0 disk.rbd2.in‚0 network.sent‚ 2.6311889e+07 disk.md0.in‚ 10.97265625 disk.rbd0.in‚0.015625 disk.md0p1.out‚ 1236.55859375) disk./.usagePercent‚83.14737225993359 disk.sdb1.in‚0# disk.nvme0n1.out‚69015.87890625% memory_percent‚39.302730560302734 proc.cpu.threads‚17 disk.md0.out‚ 1236.55859375 disk.rbd0.out‚ 3.3671875 disk.sdb2.in‚5.40625 disk.sdb.in‚5.40625# disk.nvme0n1.in‚329983.51171875! disk.sdb.out‚1283.15966796875 disk.sdb1.out‚0 disk.sda1.in‚0" disk.nvme1n1.in‚201421.6328125 network.recv‚ 4.029132e+06% disk./.usageGB‚2973.7527770996094" disk.sdb2.out‚1283.15966796875 disk.md0p1.in‚ 10.97265625 disk.sda.in‚ 5.56640625 disk.dm-0.in‚531405.14453125" disk.sda2.out‚1283.15966796875 å4ÿ(/ _wandbruntime‚11661  _runtime‚11661gqÅÇ€)Á & lr-AdamW/pg1‚8.331958815728029e-05 & lr-AdamW/pg2‚8.331958815728029e-05 trainer/global_step‚9849 $ _timestamp‚1.788475608800145e+09 _runtime‚11661.985206592 _step‚808UÚ·óÌ)Æ & lr-AdamW/pg1‚8.331958815728029e-05 & lr-AdamW/pg2‚8.331958815728029e-05 trainer/global_step‚9849 $ _timestamp‚1.788475608800145e+09 _runtime‚11661.985206592 _step‚808¨0K_4‚)/ _wandbruntime‚11661  _runtime‚11661ìF‰Dñ'ƒ)ëO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 183605843 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚71308988 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚402138315997184 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _runtime‚11661.986983651 _step‚809 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19531081971318 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚421724932913664 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 187773767 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 183605134 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚707 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 187773767 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 455843255 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19916580090368 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚421724681242112 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19586616916480 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 trainer/global_step‚9849 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 187773769 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 327142400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚425906528248320 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚425906552188416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19916883292672 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 112296146 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 455869601 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71309695 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚445823108338688 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚421724681242112 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 183605843 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚71308988 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚402138315997184 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _runtime‚11661.986983651 _step‚809 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19531081971318 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚421724932913664 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 187773767 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 183605134 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚707 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 187773767 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 455843255 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19916580090368 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚421724681242112 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19586616916480 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 trainer/global_step‚õ/Œ8 9849 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 187773769 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 327142400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚425906528248320 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚425906552188416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19916883292672 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 112296146 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 455869601 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71309695 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚445823108338688 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚421724681242112 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚402140014819412 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19918874076672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 455914594 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 187792044 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 % _timestamp‚1.7884756099579537e+09 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 183623438 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 455914594 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚421765931304960 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19588580955648 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚421766182989824 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚686 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 325031936 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚402177602034176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19588347273728 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 455888222 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚688 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 455888222 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚402177584031232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _runtime‚11663.143768748 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 187792044 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 268122550 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚445867110235136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚382607220379552 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 183622750 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 268122550 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920¹;î}'‡)úM bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19918874076672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 455914594 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 187792044 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 % _timestamp‚1.7884756099579537e+09 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 183623438 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 455914594 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚421765931304960 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19588580955648 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚421766182989824 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚686 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 325031936 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚402177602034176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19588347273728 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 455888222 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚688 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 455888222 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚402177584031232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _runtime‚11663.143768748 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 187792044 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 268122550 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚445867110235136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚382607220379552 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 183622750 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 268122550 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚402177584031232 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚402140014819412 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777ªÛIý4ˆ)/ _wandbruntime‚11663  _runtime‚11663¶—o‘V‰)Ð -loss/FBt1/assignment_loss‚0.3792925430203 1loss/SRqqt1/detection_loss‚1.8535270330806572 .loss/FBt1/detection_loss‚2.391256776120928 epoch‚8 /loss/FBt2/detection_loss‚1.7543061187392788 /loss/FRt1/assignment_loss‚2.278338872617291 /loss/FRt2/assignment_loss‚3.084048094956771 0loss/FRt3/assignment_loss‚1.2414566803334364 2loss/SRqqt2/assignment_loss‚1.5335072132533993 2loss/SRqqt3/assignment_loss‚0.9675460694035372 /loss/FRt4/assignment_loss‚3.111320622942664 2loss/SRqqt4/assignment_loss‚1.3544067113149552 .loss/FRt1/detection_loss‚2.648914785154404 /loss/FBt4/detection_loss‚2.5527106324831643 % _timestamp‚1.7884756099602342e+09 .loss/FRt2/detection_loss‚2.140222309944321 1loss/SRqqt2/detection_loss‚2.2025849480053474 0loss/SRqqt3/detection_loss‚0.825685904110567 _step‚811 0loss/FBt2/assignment_loss‚0.2176089322522528 _runtime‚11663.145389085 1loss/FBt3/assignment_loss‚0.10030847306307422 1loss/FBt4/assignment_loss‚0.15601809190896254 /loss/FRt4/detection_loss‚3.1468274268237026 .loss/FRt3/detection_loss‚1.504822016874356 0loss/SRqqt4/detection_loss‚2.466743278887964 %loss/total_loss‚40.07005276330224 trainer/global_step‚9849 2loss/SRqqt1/assignment_loss‚1.5981215399418336 /loss/FBt3/detection_loss‚0.5604776880690749OÜ(«[Š)Õ % _timestamp‚1.7884756099602342e+09 .loss/FRt2/detection_loss‚2.140222309944321 1loss/SRqqt2/detection_loss‚2.2025849480053474 0loss/SRqqt3/detection_loss‚0.825685904110567 _step‚811 0loss/FBt2/assignment_loss‚0.2176089322522528 _runtime‚11663.145389085 1loss/FBt3/assignment_loss‚0.10030847306307422 1loss/FBt4/assignment_loss‚0.15601809190896254 /loss/FRt4/detection_loss‚3.1468274268237026 .loss/FRt3/detection_loss‚1.504822016874356 0loss/SRqqt4/detection_loss‚2.466743278887964 %loss/total_loss‚40.07005276330224 trainer/global_step‚9849 2loss/SRqqt1/assignment_loss‚1.5981215399418336 /loss/FBt3/detection_loss‚0.5604776880690749 -loss/FBt1/assignment_loss‚0.3792925430203 1loss/SRqqt1/detection_loss‚1.8535270330806572 .loss/FBt1/detection_loss‚2.391256776120928 epoch‚8 /loss/FBt2/detection_loss‚1.7543061187392788 /loss/FRt1/assignment_loss‚2.278338872617291 /loss/FRt2/assignment_loss‚3.084048094956771 0loss/FRt3/assignment_loss‚1.2414566803334364 2loss/SRqqt2/assignment_loss‚1.5335072132533993 2loss/SRqqt3/assignment_loss‚0.9675460694035372 /loss/FRt4/assignment_loss‚3.111320622942664 2loss/SRqqt4/assignment_loss‚1.3544067113149552 .loss/FRt1/detection_loss‚2.648914785154404 /loss/FBt4/detection_loss‚2.5527106324831643«¿!„¼’‹)‚(:‡ àñçÔ“­Ô‡ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚140.145' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.03625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Àsç~‰Œ)‚(:þ àñçÔ«¨¾ˆ" proc.memory.rssMB‚ 3032.2265625 proc.cpu.threads‚17" disk.sdb2.out‚1284.60888671875" disk.sda2.out‚1284.60888671875 disk.sdb1.in‚0 disk.sda1.in‚0! disk.sdb.out‚1284.60888671875 disk.dm-0.in‚531405.14453125 disk.md0p1.out‚ 1237.95703125) disk./.usagePercent‚83.14285646632462 disk.sdb.in‚5.40625) proc.memory.availableMB‚ 9944.85546875 network.recv‚ 4.033195e+06 disk.md0.in‚ 10.97265625 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 3.3671875 disk.rbd2.in‚0! disk.sda.out‚1284.60888671875 disk.sda1.out‚0) proc.memory.percent‚18.50724220275879 network.sent‚ 2.6330842e+07% disk./.usageGB‚2973.5912704467773 disk.sdb2.in‚5.40625" disk.nvme1n1.in‚201421.6328125 disk.sdb1.out‚0# disk.nvme0n1.in‚329983.51171875# disk.nvme0n1.out‚69016.54296875! disk.dm-0.out‚139652.49609375! disk.nvme1n1.out‚ 70635.953125 disk.md0.out‚ 1237.95703125 disk.rbd2.out‚ 3.3671875 disk.sda.in‚ 5.56640625% memory_percent‚39.301419258117676 cpu‚25.204186745620515 disk.sda2.in‚ 5.56640625 disk.md0p1.in‚ 10.97265625XôÞ)‚(:… ïñçÔúµ‡ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚141.704' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.426 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘ÞƒŽ)‚(:ø ïñçÔÀ¤Ú‡" proc.memory.rssMB‚ 3032.2265625 disk.md0p1.out‚ 1239.78515625 disk.rbd2.in‚0 disk.sdb.out‚1286.4853515625 disk.sdb1.in‚0 disk.sdb1.out‚0" disk.nvme0n1.out‚ 69016.6484375 proc.cpu.threads‚17 network.sent‚ 2.6424938e+07 network.recv‚ 4.039998e+06 disk.sdb2.in‚5.40625 disk.sda.in‚ 5.56640625# disk.nvme0n1.in‚329983.51171875 disk.dm-0.in‚531405.14453125 disk.sda1.in‚0) disk./.usagePercent‚83.14469550721947 disk.md0.in‚ 10.97265625 disk.sda.out‚1286.4853515625! disk.dm-0.out‚139719.53515625! disk.sdb2.out‚1286.4853515625" disk.nvme1n1.in‚201421.6328125# disk.nvme1n1.out‚70702.88671875 cpu‚25.20397156917937 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 3.3671875% memory_percent‚39.302921295166016 disk.sdb.in‚5.40625 disk.sda1.out‚0! disk.sda2.out‚1286.4853515625' proc.memory.availableMB‚ 9944.609375) proc.memory.percent‚18.50724220275879 disk.rbd2.out‚ 3.3671875 disk.sda2.in‚ 5.56640625% disk./.usageGB‚2973.6570434570312 disk.md0p1.in‚ 10.97265625 disk.md0.out‚ 1239.78515625šÙ1(’)‚(:‡ þñçÔîà† gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚137.693' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.42325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªVýq)‚(:æ þñçÔ¢¨œ‡ disk.rbd2.in‚0 network.sent‚ 2.6431287e+07( disk./.usagePercent‚83.1532113907646 disk.sdb2.out‚ 1287.92578125 disk.md0.in‚ 10.97265625 disk.rbd0.in‚0.015625 disk.sdb1.in‚0# disk.nvme0n1.in‚329983.51171875 disk.dm-0.in‚531405.14453125 disk.sda2.in‚ 5.56640625 disk.sda2.out‚ 1287.92578125 disk.md0p1.out‚ 1241.17578125' proc.memory.availableMB‚ 9944.609375" proc.memory.rssMB‚ 3032.2265625 cpu‚25.203713003818017 proc.cpu.threads‚17 disk.sdb2.in‚5.40625" disk.nvme1n1.in‚201421.6328125 disk.sdb1.out‚0 disk.sda.in‚ 5.56640625) proc.memory.percent‚18.50724220275879 disk.rbd0.out‚ 3.3671875 disk.md0.out‚ 1241.17578125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb.out‚ 1287.92578125 disk.md0p1.in‚ 10.97265625% memory_percent‚39.302921295166016 disk.rbd2.out‚ 3.3671875 network.recv‚ 4.044429e+06 disk.sdb.in‚5.40625 disk.sda.out‚ 1287.92578125 disk.nvme0n1.out‚ 69109.71875 disk.dm-0.out‚ 139860.8125$ disk./.usageGB‚2973.961612701416 disk.nvme1n1.out‚ 70751.09375ñ'‚4‘)/ _wandbruntime‚11714  _runtime‚11714YNÈ’) & lr-AdamW/pg2‚7.976664300171014e-05 trainer/global_step‚9899 % _timestamp‚1.7884756610898867e+09 _runtime‚11714.274831783 _step‚812 & lr-AdamW/pg1‚7.976664300171014e-05ØÆ)Í“)Ç _step‚812 & lr-AdamW/pg1‚7.976664300171014e-05 & lr-AdamW/pg2‚7.976664300171014e-05 trainer/global_step‚9899 % _timestamp‚1.7884756610898867e+09 _runtime‚11714.274831783¬¬2Ö4”)/ _wandbruntime‚11714  _runtime‚11714f={k •)ëO F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 188687519 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19684832623104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚384446190ò¸ûÑ$853968 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 112836246 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19684598954496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚448007461857792 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚11714.275980109 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚404102599843840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚681 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚384446207893328 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚683 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 269433784 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 269433784 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20016711166976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚423787450469888 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 188687519 % _timestamp‚1.7884756610909166e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 188687517 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚404075227905154 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚423787198798336 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚404102617846784 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19684832623104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚404102617846784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 458121303 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚404074981884710 _step‚813 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 458094957 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71658770 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19684598954496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚423787450469888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚427990726750720 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19629020011826 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚423787198798336 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 269407440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚448007149395456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19628791030742 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚71658089 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 112836248 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚427990750690816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 269407440 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300D€4ö'–)ðO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚404102599843840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚681 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚384446207893328 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚683 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 269433784 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 269433784 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20016711166976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚423787450469888 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 188687519 % _timestamp‚1.7884756610909166e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 188687517 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚404075227905154 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚423787198798336 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚404102617846784 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19684832623104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚404102617846784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 458121303 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚404074981884710 _step‚813 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 458094957 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71658770 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19684598954496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚423787450469888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚427990726750720 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19629020011826 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚423787198798336 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 269407440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚448007149395456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19628791030742 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚71658089 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 112836248 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚427990750690816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 269407440 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 188687519 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19684832623104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚384446190853968 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 112836246 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19684598954496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚448007461857792 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚11714.275980109 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 184512330 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19686796662272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚9899 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 188705794 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚428032410720768 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19630749519728 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚423828448861184 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 112847050 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚384483721211392 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 458166296 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚384483738250752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19686562980352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚71665282 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 269434132 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚404141885880832 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚423828700546048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 269460502 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚448051142246912 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚11715.311881843 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 269460502 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚448050831895040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 % _timestamp‚1.7884756621259236e+09 _step‚814 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 184513032 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20018707586048 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚404114716751772 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 458139924 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚404141903883776 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚423828700546048 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚71665982 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚700 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 458166296 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 112847048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 269434132 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19686562980352 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚428032434660864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777“,&Û)úM 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 % _timestamp‚1.7884756621259236e+09 _step‚814 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 184513032 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20018707586048 WCDeviceS$Ÿ¶£=!tatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚404114716751772 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 458139924 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚404141903883776 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚423828700546048 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚71665982 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚700 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 458166296 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 112847048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 269434132 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19686562980352 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚428032434660864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚404141903883776 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 188705794 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 184512330 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19686796662272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚9899 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 188705794 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚428032410720768 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19630749519728 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚423828448861184 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 112847050 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚384483721211392 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 458166296 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚384483738250752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19686562980352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚71665282 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 269434132 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚404141885880832 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚423828700546048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 269460502 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚448051142246912 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚11715.311881843 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 269460502 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚448050831895040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 270771736 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚425849716354560 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚688 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚406049683816862 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚450191295481344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚425849968026112 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19782814661120 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19782814661120 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚450191612137984 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚386322708725168 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 189601267 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 189601267 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 185380568 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 113376348 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚406049437796418 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚425849716354560 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 460346659 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 270771736 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 185381258 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19726729071250 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 # _timestamp‚1.78847571452907e+09 _step‚817 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 189601269 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚690 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20116370228224 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚406066901693440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 460373005 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544ëbzWRª)îO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚690 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20116370228224 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚406066901693440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 460373005 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 270771736 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚425849716354560 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚688 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚406049683816862 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚450191295481344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚425849968026112 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19782814661120 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19782814661120 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚450191612137984 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚386322708725168 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 189601267 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 189601267 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 185380568 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 113376348 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚406049437796418 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚425849716354560 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 460346659 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 270771736 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 185381258 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19726729071250 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 # _timestamp‚1.78847571452907e+09 _step‚817 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 189601269±'ŠP4«)/ _wandbruntime‚11768  _runtime‚11768 ñóýù&¬)óM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 460417998 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚406106205733376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72011580 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚425891218102272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 270772084 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19784778686976 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19785012368896 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19785012368896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚386360256121952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚425891218102272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚684 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚406089172663480 _runtime‚11768.678844459 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20118365023232 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚406106205733376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚72010898 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 113387150 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 460391626 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 460391626 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚430116609223168 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚406106187730432 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19784778686976 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 185398046 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚425890966417408 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚450234974246400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 189619542 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚9949 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19728916541528 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 189619542 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 460417998 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚430116633163264 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚386360239082592 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 270798454 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚406106187730432 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 270798454 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 270772084 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 185398730 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 # _timestamp‚1.78847571549346e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496ŠeÔµþ&­)øM XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚406106205733376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚72010898 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 113387150 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 460391626 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 460391626 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚430116609223168 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚406106187730432 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19784778686976 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 185398046 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚425890966417408 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚450234974246400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 189619542 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚9949 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19728916541528 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 189619542 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 460417998 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚430116633163264 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚386360239082592 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 270798454 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚406106187730432 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 270798454 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 270772084 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 185398730 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 # _timestamp‚1.78847571549346e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 460417998 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚406106205733376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72011580 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚425891218102272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 270772084 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19784778686976 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19785012368896 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19785012368896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚386360256121952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚425891218102272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚684 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚406089172663480 _runtime‚11768.678844459 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20118365023232 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249²z»wT4®)/ _wandbruntime‚11768  _runtime‚11768Øî«^¯)Ø .loss/FRt3/detection_loss‚1.479994453600983 1loss/SRqqt2/detection_loss‚2.2640163328717735 0loss/FRt4/assignment_loss‚2.5162199464711277 2loss/SRqqt1/assignment_loss‚1.4085881801821836 1loss/FBt3/assignment_loss‚0.11876272134783539 0loss/SRqqt4/detection_loss‚2.716457518438498 /loss/FBt2/detection_loss‚1.8678011617490224 epoch‚8 0loss/FBt1/assignment_loss‚0.5501493820435522 2loss/FBt4/assignment_loss‚0.015806354737529362 %loss/total_loss‚40.05918571235511 /loss/FRt2/assignment_loss‚3.263436550789691 /loss/FBt4/detection_loss‚2.2268575959735446 1loss/FBt2/assignment_loss‚0.12330984158454582 /loss/FRt1/detection_loss‚2.7101019009163507 /loss/FRt4/detection_loss‚3.0689494664018806 1loss/SRqqt1/detection_loss‚1.9069934247470484 $ _timestamp‚1.788475715495102e+09 2loss/SRqqt4/assignment_loss‚1.3238863005923729 .loss/FBt3/detection_loss‚0.482908673251846 trainer/global_step‚9949 _runtime‚11768.679788633 2loss/SRqqt2/assignment_loss‚1.4356163475005066 2loss/SRqqt3/assignment_loss‚1.0450329689026063 /loss/FRt2/detection_loss‚2.1079366143846743 1loss/SRqqt3/detection_loss‚0.6751153595822373 .loss/FBt1/detection_loss‚2.606014837821325 _step‚819 0loss/FRt1/assignment_loss‚2.9307885546433297 0loss/FRt3/assignment_loss‚1.2144412238206437vûc°)Ý /loss/FRt1/detection_loss‚2.7101019009163507 /loss/FRt4/detection_loss‚3.0689494664018806 1loss/SRqqt1/detection_loss‚1.9069934247470484 $ _timestamp‚1.788475715495102e+09 2loss/SRqqt4/assignment_loss‚1.3238863005923729 .loss/FBt3/detection_loss‚0.482908673251846 trainer/global_step‚9949 _runtime‚11768.679788633 2loss/SRqqt2/assignment_loss‚1.4356163475005066 2loss/SRqqt3/assignment_loss‚1.0450329689026063 /loss/FRt2/detection_loss‚2.1079366143846743 1loss/SRqqt3/detection_loss‚0.6751153595822373 .loss/FBt1/detection_loss‚2.606014837821325 _step‚819 0loss/FRt1/assignment_loss‚2.9307885546433297 0loss/FRt3/assignment_loss‚1.2144412238206437 .loss/FRt3/detection_loss‚1.479994453600983 1loss/SRqqt2/detection_loss‚2.2640163328717735 0loss/FRt4/assignment_loss‚2.5162199464711277 2loss/SRqqt1/assignment_loss‚1.4085881801821836 1loss/FBt3/assignment_loss‚0.11876272134783539 0loss/SRqqt4/detection_loss‚2.716457518438498 /loss/FBt2/detection_loss‚1.8678011617490224 epoch‚8 0loss/FBt1/assignment_loss‚0.5501493820435522 2loss/FBt4/assignment_loss‚0.015806354737529362 %loss/total_loss‚40.05918571235511 /loss/FRt2/assignment_loss‚3.263436550789691 /loss/FBt4/detection_loss‚2.2268575959735446 1loss/FBt2/assignment_loss‚0.12330984158454582³ÃÞk›±)‚(:… ÉòçÔŠñ‰‡ gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚130.232' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.558 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿBÔ)|²)‚(:ñ Éòçԟ╈ disk.sda2.out‚1296.263671875 disk.md0.out‚ 1249.20703125 disk.sda2.in‚ 5.56640625% memory_percent‚40.349388122558594 cpu‚25.2026054704197( disk./.usagePercent‚83.1568207831716 disk.sdb2.in‚5.40625" disk.nvme1n1.in‚201421.6328125 disk.md0p1.in‚ 10.97265625) proc.memory.percent‚18.50724220275879 proc.cpu.threads‚17% disk./.usageGB‚2974.0907020568848 disk.sdb2.out‚1296.263671875 disk.rbd0.out‚ 3.3671875 disk.sdb1.in‚0 disk.sdb1.out‚0# disk.nvme0n1.in‚329983.51171875 network.sent‚ 2.6642301e+07 network.recv‚ 4.070223e+06 disk.md0.in‚ 10.97265625 disk.rbd0.in‚0.015625& proc.memory.availableMB‚ 9773.15625# disk.nvme1n1.out‚71158.21484375 disk.dm-0.in‚531405.14453125! disk.dm-0.out‚140507.06640625 disk.rbd2.in‚0 disk.sdb.out‚1296.263671875 disk.md0p1.out‚ 1249.20703125 disk.sda1.out‚0 disk.sda.out‚1296.263671875" disk.nvme0n1.out‚ 69348.8515625 disk.sda1.in‚0" proc.memory.rssMB‚ 3032.2265625 disk.rbd2.out‚ 3.3671875 disk.sdb.in‚5.40625 disk.sda.in‚ 5.56640625ñGÆᑳ)‚(:† ØòçÔÎÑχ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.357' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.83925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'@Òx´)‚(:í ØòçÔÞýä‡! disk.sdb2.out‚1298.2236328125( disk./.usagePercent‚83.1673783626226$ disk./.usageGB‚2974.468292236328 disk.sdb1.in‚0 disk.sda.in‚ 5.56640625 disk.sda1.in‚0) proc.memory.percent‚18.50724220275879 disk.rbd2.out‚ 3.3671875" disk.nvme1n1.in‚201421.6328125 disk.sdb.in‚5.40625 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 3.3671875 cpu‚25.202375040454278! disk.nvme0n1.out‚ 69540.734375 disk.md0p1.in‚ 10.97265625 disk.md0p1.out‚ 1251.09375% memory_percent‚40.347838401794434" proc.memory.rssMB‚ 3032.2265625 disk.sda.out‚1298.2236328125# disk.nvme0n1.in‚329983.51171875 disk.sda1.out‚0 disk.sda2.in‚ 5.56640625) proc.memory.availableMB‚ 9773.41015625 network.recv‚ 4.076254e+06! disk.nvme1n1.out‚ 71158.359375 disk.dm-0.in‚531405.14453125 disk.sdb1.out‚0 disk.dm-0.out‚ 140699.09375! disk.sda2.out‚1298.2236328125 disk.sdb2.in‚5.40625 disk.md0.in‚ 10.97265625 disk.md0.out‚ 1251.09375 network.sent‚ 2.6749113e+07 disk.rbd2.in‚0 disk.sdb.out‚1298.2236328125 proc.cpu.threads‚17!{˜€‘µ)‚(:† çòçԡ츆 gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚127.446' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.8615 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ù…Æ!‚¶)‚(:÷ çòçÔÇËû‡ disk.rbd2.out‚ 3.3671875 disk.sda.out‚1299.7666015625# disk.nvme0n1.in‚329983.51171875 disk.sda1.out‚0 disk.sda2.in‚ 5.56640625% disk./.usageGB‚2974.5920066833496 disk.md0p1.out‚ 1252.578125) proc.memory.availableMB‚ 9773.40234375 proc.cpu.threads‚17 network.sent‚ 2.6754318e+07 disk.sdb.out‚1299.7666015625# disk.nvme0n1.out‚69750.65234375 network.recv‚ 4.079962e+06) disk./.usagePercent‚83.17083747033985 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 3.3671875 disk.sda.in‚ 5.56640625! disk.dm-0.out‚140909.16015625" proc.memory.rssMB‚ 3032.2265625 cpu‚25.202156355113065" disk.nvme1n1.in‚201421.6328125 disk.sdb.in‚5.40625 disk.dm-0.in‚531405.14453125! disk.sda2.out‚1299.7666015625 disk.md0p1.in‚ 10.97265625! disk.sdb2.out‚1299.7666015625) proc.memory.percent‚18.50724220275879" disk.nvme1n1.out‚ 71158.5078125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0.in‚ 10.97265625 disk.rbd2.in‚0 disk.sdb2.in‚5.40625 disk.md0.out‚ 1252.578125 disk.sda1.in‚0% memory_percent‚40.347886085510254îW ·)‚(:… öòçԌޙ† gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚121.536' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.384 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²™è¸)‚(:õ öòçÔöÊœ‡ disk.md0p1.out‚ 1253.9296875 cpu‚25.201945949931204# disk.nvme1n1.out‚71158.58984375 disk.sda.in‚ 5.56640625 disk.nvme0n1.out‚ 69982.71875! disk.dm-0.out‚141141.30859375 disk.sda1.out‚0! disk.sda2.out‚1301.1708984375 network.recv‚ 4.08458e+06 disk.rbd2.in‚0 disk.rbd2.out‚ 3.3671875 disk.sdb.in‚5.40625 disk.sda.out‚1301.1708984375# disk.nvme0n1.in‚329983.51171875 disk.sda1.in‚0 disk.md0p1.in‚ 10.97265625% disk./.usageGB‚2974.5920639038086„gØœc disk.md0.in‚ 10.97265625 disk.md0.out‚ 1253.9296875$ memory_percent‚40.34779071807861) proc.memory.availableMB‚ 9773.41796875 proc.cpu.threads‚17" disk.nvme1n1.in‚201421.6328125 disk.sdb1.out‚0 disk.sdb2.in‚5.40625" proc.memory.rssMB‚ 3032.2265625) disk./.usagePercent‚83.17083907024782 disk.sdb1.in‚0 disk.sda2.in‚ 5.56640625 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 3.3671875) proc.memory.percent‚18.50724220275879 network.sent‚ 2.6760676e+07! disk.sdb2.out‚1301.1708984375 disk.sdb.out‚1301.1708984375 disk.dm-0.in‚531405.14453125òÖ4¹)/ _wandbruntime‚11823  _runtime‚11823<`…Ǻ)Á _runtime‚11823.485541987 _step‚820 & lr-AdamW/pg1‚7.286933099488936e-05 & lr-AdamW/pg2‚7.286933099488936e-05 trainer/global_step‚9999 $ _timestamp‚1.788475770300586e+09¶(?ÅÌ»)Æ & lr-AdamW/pg1‚7.286933099488936e-05 & lr-AdamW/pg2‚7.286933099488936e-05 trainer/global_step‚9999 $ _timestamp‚1.788475770300586e+09 _runtime‚11823.485541987 _step‚820´Jà ‚4¼)/ _wandbruntime‚11823  _runtime‚11823•:Çð'½)êO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚427912233910784 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚72349267 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 _step‚821 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚427912485582336 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 190515017 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19824667111758 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19881030367744 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚427912485582336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚427912233910784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚408031203543040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19824896092842 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 272109688 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.788475770301434e+09 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚388199243635728 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19881264036352 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 186265713 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚703 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚701 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 272083344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 462598361 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19881030367744 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 113916448 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚452375765886976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚72349968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20216327571968 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 190515019 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚408031203543040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚432159123755520 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚452375451327488 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚408024139728570 _runtime‚11823.486757502 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19881264036352 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19881264036352 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 _step‚821 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚427912485582336 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 190515017 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19824667111758 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19881030367744 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚427912485582336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚427912233910784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚408031203543040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19824896092842 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 272109688 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.788475770301434e+09 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚388199243635728 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19881264036352 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 186265713 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚703 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚701 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 272083344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 462598361 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19881030367744 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 113916448 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚452375765886976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚72349968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20216327571968µE‘ð4¿)/ _wandbruntime‚11824  _runtime‚11824æÄ´˜ú&À)ôM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19882994393600 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19882994393600 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚690 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚408063628575188 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20218319052288 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 462643328 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _runtime‚11824.486284058 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚408070507582976 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 272136406 $ _timestamp‚1.788475771300407e+09 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19826625600744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 186284165 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 462643328 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚822 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 186283473 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚408070507582976 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚692 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 trainer/global_step‚9999 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 190533292 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 113927248 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚388236773993152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 272136406 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚427953483973632 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19883228075520 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 462669700 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 190533294 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 113927250 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 190533294 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚408070489580032 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20218609658368 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚408070489580032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19826854582036 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 272110036 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 190533292 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72356915 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚72356225 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚432200831665664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚427953735658496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 462669700 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚452419126777856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 272110036 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚432200807725568öe1 IÁ)ùM 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 190533294 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚408070489580032 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20218609658368 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚408070489580032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19826854582036 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 272110036 MDDeviceStatsMonitor.on_trai„ç´Â¶"n_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 190533292 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72356915 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚72356225 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚432200831665664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚427953735658496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 462669700 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚452419126777856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 272110036 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚432200807725568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19882994393600 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19882994393600 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚690 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚408063628575188 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20218319052288 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 462643328 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _runtime‚11824.486284058 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚408070507582976 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 272136406 $ _timestamp‚1.788475771300407e+09 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19826625600744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 186284165 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 462643328 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚822 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 186283473 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚408070507582976 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚692 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 trainer/global_step‚9999 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 190533292 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 113927248 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚388236773993152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 272136406 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚427953483973632 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19883228075520 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 462669700 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 190533294 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 113927250¶$u8*4Â)/ _wandbruntime‚11824  _runtime‚11824¿Æ'WÃ)Ñ -loss/FBt4/detection_loss‚2.09143336613973 %loss/total_loss‚39.31554516565963 _runtime‚11824.487992775 /loss/FRt1/assignment_loss‚2.459384814781301 /loss/FBt4/assignment_loss‚0.380404167125622 -loss/FRt4/detection_loss‚2.97050201009821 0loss/SRqqt2/detection_loss‚2.560505350828171 2loss/SRqqt3/assignment_loss‚0.8928148199068873 2loss/SRqqt4/assignment_loss‚1.3781852240745838 trainer/global_step‚9999 1loss/SRqqt1/assignment_loss‚1.196631545708938 1loss/FBt2/assignment_loss‚0.24690853077393182 /loss/FRt3/detection_loss‚1.1406637969668265 0loss/FRt4/assignment_loss‚2.5122965817098266 _step‚823 1loss/SRqqt3/detection_loss‚0.7754453735916238 2loss/SRqqt2/assignment_loss‚1.3089456853270531 1loss/FBt1/assignment_loss‚0.06859475195067688 /loss/FRt2/detection_loss‚1.9771449066687463 .loss/FBt1/detection_loss‚2.845391482114792 /loss/FBt2/detection_loss‚1.9276747480034828 /loss/FBt3/detection_loss‚0.5206729857517141 epoch‚8 % _timestamp‚1.7884757713032153e+09 0loss/FRt3/assignment_loss‚1.1518378949459689 0loss/FRt2/assignment_loss‚3.6056673569338664 1loss/FBt3/assignment_loss‚0.03980936295298925 -loss/FRt1/detection_loss‚2.73866968645769 1loss/SRqqt1/detection_loss‚2.0768273587931287 0loss/SRqqt4/detection_loss‚2.449133364053873±íÐë\Ä)Ö /loss/FBt2/detection_loss‚1.9276747480034828 /loss/FBt3/detection_loss‚0.5206729857517141 epoch‚8 % _timestamp‚1.7884757713032153e+09 0loss/FRt3/assignment_loss‚1.1518378949459689 0loss/FRt2/assignment_loss‚3.6056673569338664 1loss/FBt3/assignment_loss‚0.03980936295298925 -loss/FRt1/detection_loss‚2.73866968645769 1loss/SRqqt1/detection_loss‚2.0768273587931287 0loss/SRqqt4/detection_loss‚2.449133364053873 -loss/FBt4/detection_loss‚2.09143336613973 %loss/total_loss‚39.31554516565963 _runtime‚11824.487992775 /loss/FRt1/assignment_loss‚2.459384814781301 /loss/FBt4/assignment_loss‚0.380404167125622 -loss/FRt4/detection_loss‚2.97050201009821 0loss/SRqqt2/detection_loss‚2.560505350828171 2loss/SRqqt3/assignment_loss‚0.8928148199068873 2loss/SRqqt4/assignment_loss‚1.3781852240745838 trainer/global_step‚9999 1loss/SRqqt1/assignment_loss‚1.196631545708938 1loss/FBt2/assignment_loss‚0.24690853077393182 /loss/FRt3/detection_loss‚1.1406637969668265 0loss/FRt4/assignment_loss‚2.5122965817098266 _step‚823 1loss/SRqqt3/detection_loss‚0.7754453735916238 2loss/SRqqt2/assignment_loss‚1.3089456853270531 1loss/FBt1/assignment_loss‚0.06859475195067688 /loss/FRt2/detection_loss‚1.9771449066687463 .loss/FBt1/detection_loss‚2.845391482114792·Y¯Ã0’Å)‚(:‡ …óçÔ»Û•‡ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚140.145' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.03625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06ô8JÆ)‚(:ö …óçÔ›šÐ‡# disk.nvme1n1.out‚71158.96484375 disk.sdb.out‚1303.1904296875 disk.sdb1.out‚0 disk.sda1.out‚0$ memory_percent‚40.35220146179199" proc.memory.rssMB‚ 3032.2265625 cpu‚25.201731074440865 disk.sdb1.in‚0# disk.nvme0n1.in‚329983.51171875 disk.dm-0.out‚ 141240.71875 disk.sda2.in‚ 5.56640625! disk.sdb2.out‚1303.1904296875) proc.memory.percent‚18.50724220275879 network.sent‚ 2.6766543e+07 disk.dm-0.in‚531405.14453125! disk.sda2.out‚1303.1904296875 disk.md0.in‚ 10.97265625) disk./.usagePercent‚83.18487410302758" disk.nvme1n1.in‚201421.6328125# disk.nvme0n1.out‚70101.75390625 disk.md0.out‚ 1255.88671875( proc.memory.availableMB‚ 9772.6953125 disk.sdb.in‚5.40625 disk.sda.out‚1303.1904296875 disk.md0p1.in‚ 10.97265625 disk.sdb2.in‚5.40625 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 network.recv‚ 4.088686e+06$ disk./.usageGB‚2975.094024658203 disk.sda1.in‚0 disk.md0p1.out‚ 1255.88671875 disk.rbd0.out‚ 3.3671875 disk.rbd2.out‚ 3.3671875 proc.cpu.threads‚17 disk.sda.in‚ 5.56640625'×üv›Ç)‚(: ”óçÔ˜¼’‡ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚107.896' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚26.973999999999997 gpu.0.smClock‚1260 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÅpûŽsÈ)‚(:è ”óçÔ§âèˆ disk.sda2.in‚ 5.56640625" disk.nvme1n1.in‚201421.6328125 network.recv‚ 4.095856e+06 disk.md0.out‚ 1257.3359375 disk.rbd2.out‚ 3.3671875 disk.sdb1.in‚0 disk.dm-0.in‚531405.14453125 disk.sda1.out‚0% disk./.usageGB‚2974.8149032592773 disk.sdb2.in‚5.40625 disk.rbd0.in‚0.015625 disk.sdb.out‚ 1304.6953125 disk.dm-0.out‚ 141409.03125 cpu‚25.201513270441065 disk.sdb2.out‚ 1304.6953125 disk.md0.in‚ 10.97265625 disk.rbd2.in‚0 disk.md0p1.in‚ 10.97265625 disk.md0p1.out‚ 1257.3359375" proc.memory.rssMB‚ 3032.2265625) proc.memory.percent‚18.50724220275879) disk./.usagePercent‚83.17706975189225 disk.sda.in‚ 5.56640625% memory_percent‚40.351295471191406# disk.nvme1n1.out‚71159.12890625 disk.sda.out‚ 1304.6953125# disk.nvme0n1.out‚70249.90234375 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda2.out‚ 1304.6953125 network.sent‚ 2.6876442e+07 disk.rbd0.out‚ 3.3671875 disk.sdb1.out‚0# disk.nvme0n1.in‚329983.51171875& proc.memory.availableMB‚ 9772.84375 disk.sdb.in‚5.40625$¸2ãÉ)‚(:… £óçÔÍÝʇ gpu.0.gpu‚7.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚140.754' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.1885 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Sœ‰x…Ê)‚(:ú £óçÔö„Þˆ network.recv‚ 4.099451e+06 disk.dm-0.in‚531405.14453125! disk.dm-0.out‚141520.38671875 disk.sda1.out‚0" disk.sda2.out‚1306.15380859375 disk.rbd0.out‚ 3.3671875 disk.rbd2.out‚ 3.3671875" proc.memory.rssMB‚ 3032.2265625 disk.md0p1.in‚ 10.97265625 disk.sdb2.in‚5.40625 disk.md0.in‚ 10.97265625 disk.sdb1.in‚0 disk.sdb1.out‚0) proc.memory.percent‚18.50724220275879 disk.md0.out‚ 1258.73046875! disk.nvme1n1.out‚ 71159.234375! disk.sda.out‚1306.15380859375 disk.rbd0.in‚0.015625" disk.nvme1n1.in‚201421.6328125! disk.sdb.out‚1306.15380859375 disk.sda1.in‚0" disk.sdb2.out‚1306.15380859375 disk.sdb.in‚5.40625 disk.sda.in‚ 5.56640625& proc.memory.availableMB‚ 9773.09375) disk./.usagePercent‚83.17778512408118 disk.sda2.in‚ 5.56640625# disk.nvme0n1.in‚329983.51171875% memory_percent‚40.349769592285156$ disk./.usageGB‚2974.840488433838 disk.rbd2.in‚0# disk.nvme0n1.out‚70361.15234375 cpu‚25.201301592251905 network.sent‚ 2.6881747e+07 disk.md0p1.out‚ 1258.73046875 proc.cpu.threads‚17ôÚç4Ë)/ _wandbruntime‚11879  _runtime‚11879²”2 ÉÌ)à & lr-AdamW/pg1‚6.952640153312303e-05 & lr-AdamW/pg2‚6.952640153312303e-05 trainer/global_step‚10049 % _timestamp‚1.7884758263034258e+09 _runtime‚11879.488497764 _step‚824Ãx…ÎÍ)È & lr-AdamW/pg1‚6.952640153312303e-05 & lr-AdamW/pg2‚6.952640153312303e-05 trainer/global_step‚10049 % _timestamp‚1.7884758263034258e+09 _runtime‚11879.488497764 _step‚824¸\Åý³4Î)/ _wandbruntime‚11879  _runtime‚11879ãÓ9<ò'Ï)ìO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 273447640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚409995505392640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19922834133350 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 191428767 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19922605152266 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚409998595640278 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19979479742976 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚429974751467008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚72694479 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚688 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚454559583311360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚434243346198016 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 273447640 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 114456546 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 273421296 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 464850063 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 191428769 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20316553769984 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚409995523395584 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19979246074368 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 187151715 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 187151025 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚72695167 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7884758263043408e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20316261053440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚825 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19979246074368 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚409998349619834 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 114456548 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 191428767 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 464876409 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 273421296 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚409995505392640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚429975003138560 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 trainer/global_step‚10049 ~–ÌÐ)ñO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚429975003138560 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 trainer/global_step‚10049 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 273447640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚409995505392640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19922834133350 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 191428767 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19922605152266 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚409998595640278 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19979479742976 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚429974751467008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚72694479 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚688 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚454559583311360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚434243346198016 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 273447640 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 114456546 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 273421296 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 464850063 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 191428769 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20316553769984 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚409995523395584 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19979246074368 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 187151715 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 187151025 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚72695167 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7884758263043408e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20316261053440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚825 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19979246074368 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚409998349619834 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 114456548 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 191428767 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 464876409 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 273421296 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚409995505392640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309¹ÓW¾@‘Ñ)‚(:† ²óçÔßÄî† gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚129.282' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.3205 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VÍ0Ò)‚(:ô ²óçÔÔÍ̇ disk.sdb1.out‚0 disk.dm-0.in‚531405.14453125 network.sent‚ 2.688593e+07 network.recv‚ 4.104186e+06! disk.sdb2.out‚1307.7587890625 disk.rbd2.in‚0 disk.sdb.in‚5.40625 disk.sdb1.in‚0 disk.md0p1.out‚ 1260.265625( proc.memory.availableMB‚ 9773.1015625 cpu‚25.201032205930908) disk./.usagePercent‚83.17958427393701% disk./.usageGB‚2974.9048347473145 disk.sdb.out‚1307.7587890625 disk.md0p1.in‚ 10.97265625% memory_percent‚40.349721908569336 disk.md0.in‚ 10.97265625# disk.nvme0n1.in‚329983.51171875 disk.sda1.in‚0 disk.sda.out‚1307.7587890625! disk.dm-0.out‚141733.21484375 disk.sda1.out‚0 disk.rbd2.out‚ 3.3671875 disk.rbd0.out‚ 3.3671875) proc.memory.percent‚18.50724220275879 proc.cpu.threads‚17 disk.rbd0.in‚0.015625! disk.nvme1n1.out‚ 71159.328125" proc.memory.rssMB‚ 3032.2265625" disk.nvme1n1.in‚201421.6328125 disk.md0.out‚ 1260.265625 disk.sda.in‚ 5.56640625# disk.nvme0n1.out‚70573.88671875! disk.sda2.out‚1307.7587890625 disk.sdb2.in‚5.40625 disk.sda2.in‚ 5.56640625ä@š4Ó)/ _wandbruntime‚11880  _runtime‚11880ñÓ{–ü&Ô)öM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 464895030 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 273474358 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚434285030168064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20318260168704 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 191447044 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚454603266396672 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19924563641252 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚826 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚410034791429632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚410037838466244 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚430016253214720 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚410034809432576 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚434285006227968 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 187170076 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 187169415 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 273474358 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚410034791429632 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚454603572554240 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19924792622544 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚430016253214720 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚410034809432576 % _timestamp‚1.7884758274334605e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 273447988 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19981210100224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19981210100224 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19981443782144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚72702067 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 _runtime‚11880.619784611 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 464921402 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚430016001529856 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚659 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚390113291864352 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚390113274824992 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 191447042 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚661 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20318542386176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19981443782144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 464921402 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 114467348 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 trainer/global_step‚10049 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 114467350 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 191447044 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚410038084486896 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19924563641252 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚826 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚410034791429632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚410037838466244 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚430016253214720 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚410034809432576 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚434285006227968 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 187170076 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 187169415 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 273474358 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚410034791429632 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚454603572554240 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19924792622544 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚430016253214720 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚410034809432576 % _timestamp‚1.7884758274334605e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 273447988 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19981210100224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19981210100224 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19981443782144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚72702067 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 _runtime‚11880.619784611 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 464921402 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚430016001529856 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚659 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚390113291864352 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚390113274824992 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 191447042 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚661 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20318542386176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19981443782144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 464921402 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 114467348 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 trainer/global_step‚10049 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 114467350 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 191447044 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚410038084486896 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 273474358 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚434285030168064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20318260168704 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273º¹q½Ù4Ö)/ _wandbruntime‚11880  _runtime‚11880*f?^×)Ø 0loss/FRt2/assignment_loss‚3.0288747906684876 1loss/FBt2/assignment_loss‚0.05133460038751636 /loss/FRt1/detection_loss‚2.6467330343014486 /loss/FBt4/detection_loss‚2.4100255221128464 % _timestamp‚1.7884758274363215e+09 1loss/FBt1/assignment_loss‚0.27919753603595865 0loss/FBt4/assignment_loss‚0.5364336119091462 /loss/FRt2/detection_loss‚1.9688061837636104 .loss/FRt4/detection_loss‚2.712448313832283 0loss/FRt3/assignment_loss‚1.4410795470078785 2loss/SRqqt4/assignment_loss‚1.9171671495358977 _runtime‚11880.621159526 &loss/total_loss‚41.312597536980526 epoch‚8 _step‚827 2loss/FBt3/assignment_loss‚0.028572107368909107 /loss/FBt2/detection_loss‚2.1786076898376145 /loss/FBt3/detection_loss‚0.5736835372785257 trainer/global_step‚10049 /loss/FRt1/assignment_loss‚2.696614645622872 /loss/FRt4/assignment_loss‚3.203923385590315 1loss/SRqqt3/detection_loss‚1.0668794708671394 1loss/SRqqt4/detection_loss‚2.4829898785661766 2loss/SRqqt1/assignment_loss‚1.4245898570451472 1loss/SRqqt2/assignment_loss‚1.516448494190207 0loss/SRqqt3/assignment_loss‚1.08081177075359 /loss/FRt3/detection_loss‚1.3811363188944983 1loss/SRqqt1/detection_loss‚2.1871552674306765 0loss/SRqqt2/detection_loss‚1.943423704459117 .loss/FBt1/detection_loss‚2.555661119520664¥]³ cØ)Ý /loss/FRt1/assignment_loss‚2.696614645622872 /loss/FRt4/assignment_loss‚3.203923385590315 1loss/SRqqt3/detection_loss‚1.0668794708671394 1loss/SRqqt4/detection_loss‚2.4829898785661766 2loss/SRqqt1/assignment_loss‚1.4245898570451472 1loss/SRqqt2/assignment_loss‚1.516448494190207 0loss/SRqqt3/assignment_loss‚1.08081177075359 /loss/FRt3/detection_loss‚1.3811363188944983 1loss/SRqqt1/detection_loss‚2.1871552674306765 0loss/SRqqt2/detection_loss‚1.943423704459117 .loss/FBt1/detection_loss‚2.555661119520664 0loss/FRt2/assignment_loss‚3.0288747906684876 1loss/FBt2/assignment_loss‚0.05133460038751636 /loss/FRt1/detection_loss‚2.6467330343014486 /loss/FBt4/detection_loss‚2.4100255221128464 % _timestamp‚1.7884758274363215e+09 1loss/FBt1/assignment_loss‚0.27919753603595865 0loss/FBt4/assignment_loss‚0.5364336119091462 /loss/FRt2/detection_loss‚1.9688061837636104 .loss/FRt4/detection_loss‚2.712448313832283 0loss/FRt3/assignment_loss‚1.4410795470078785 2loss/SRqqt4/assignment_loss‚1.9171671495358977 _runtime‚11880.621159526 &loss/total_loss‚41.312597536980526 epoch‚8 _step‚827 2loss/FBt3/assignment_loss‚0.028572107368909107 /loss/FBt2/detection_loss‚2.1786076898376145 /loss/FBt3/detection_loss‚0.5736835372785257 trainer/global_step‚10049»í¶>Ù)‚(:… ÁóçÔÌ﬌ gpu.0.gpu‚3.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚145.438' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.3595 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê ºý…Ú)‚(:ú ÁóçÔö™ý network.sent‚ 2.6982245e+07 disk.rbd2.out‚ 3.3671875 disk.sda.in‚ 5.56640625! disk.sda.out‚1309.61474609375! disk.dm-0.out‚141971.55078125 disk.sda1.in‚0 disk.md0p1.in‚ 10.97265625 disk.md0p1.out‚ 1262.05078125" disk.sdb2.out‚1309.61474609375 disk.md0.in‚ 10.97265625 disk.sdb1.in‚0" disk.sda2.out‚1309.61474609375" disk.nvme1n1.in‚201421.6328125 disk.sdb.in‚5.40625 cpu‚25.20084714200329 disk.md0.out‚ 1262.05078125 disk.rbd0.out‚ 3.3671875 disk.rbd2.in‚0 disk.sdb1.out‚0 network.recv‚ 4.110618e+06# disk.nvme1n1.out‚71159.53515625 disk.rbd0.in‚0.015625) disk./.usagePercent‚83.18481778626663$ disk./.usageGB‚2975.092010498047 disk.sdb2.in‚5.40625! disk.sdb.out‚1309.61474609375# disk.nvme0n1.in‚329983.51171875) proc.memory.percent‚18.50724220275879 disk.dm-0.in‚531405.14453125$ memory_percent‚40.35000801086426( proc.memory.availableMB‚ 9773.0546875 proc.cpu.threads‚17 disk.sda2.in‚ 5.56640625! disk.nvme0n1.out‚ 70812.015625 disk.sda1.out‚0" proc.memory.rssMB‚ 3032.2265625àVH‘Û)‚(:† ÐóçÔä×›ˆ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.357' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.83925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Î̿΂Ü)‚(:÷ ÐóçÔ蔲‰ network.recv‚ 4.117297e+06 disk.sdb1.out‚0 disk.rbd0.out‚ 3.3671875 disk.sdb1.in‚0 disk.dm-0.in‚531405.14453125 disk.sda1.out‚0 disk.sda2.in‚ 5.56640625 disk.md0p1.in‚ 10.97265625 disk.sdb.in‚5.40625! disk.sdb.out‚1311.08056640625# disk.nvme0n1.in‚329983.51171875" disk.sda2.out‚1311.08056640625 disk.md0p1.out‚ 1263.4609375 cpu‚25.200644891383632" disk.sdb2.out‚1311.08056640625 disk.md0.out‚ 1263.4609375 disk.rbd2.in‚0 disk.rbd2.out‚ 3.3671875" disk.nvme1n1.out‚ 71159.6640625$ disk./.usageGB‚2975.092010498047 disk.sda.in‚ 5.56640625 disk.dm-0.out‚142133.3828125 disk.rbd0.in‚0.015625" disk.nvme1n1.in‚201421.6328125" proc.memory.rssMB‚ 3032.2265625 proc.cpu.threads‚17 network.sent‚ 2.7081725e+07) disk./.usagePercent‚83.18481778626663 disk.sda1.in‚0 disk.sdb2.in‚5.40625 disk.md0.in‚ 10.97265625$ memory_percent‚40.34850597381592) proc.memory.percent‚18.50724220275879! disk.sda.out‚1311.08056640625 disk.nvme0n1.out‚ 70973.71875) proc.memory.availableMB‚ 9773.30078125ZŃü‘Ý)‚(:† ßóçÔªþƒˆ gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚101.986' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.4965 gpu.0.smClock‚1185 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½]_q}Þ)‚(:ê ßóçԳ급 disk.md0p1.in‚ 10.97265625 disk.sda.out‚ 1312.7109375) disk./.usagePercent‚83.184818426&Žø22982% disk./.usageGB‚2975.0920333862305" disk.nvme1n1.out‚ 71159.8203125$ memory_percent‚40.34848213195801 cpu‚25.200434787807716 disk.sdb.in‚5.40625( proc.memory.availableMB‚ 9773.3046875 network.sent‚ 2.7087043e+07 network.recv‚ 4.121079e+06" disk.nvme1n1.in‚201421.6328125 disk.dm-0.out‚142392.4609375 disk.md0.in‚ 10.97265625) proc.memory.percent‚18.50724220275879 disk.sda.in‚ 5.56640625 disk.sda2.in‚ 5.56640625" proc.memory.rssMB‚ 3032.2265625 disk.sda1.out‚0 disk.sdb1.out‚0 disk.sdb.out‚ 1312.7109375 disk.sdb1.in‚0! disk.nvme0n1.out‚ 71232.640625# disk.nvme0n1.in‚329983.51171875 disk.dm-0.in‚531405.14453125 disk.sda2.out‚ 1312.7109375 disk.sdb2.in‚5.40625 disk.md0.out‚ 1265.02734375 disk.rbd0.out‚ 3.3671875 disk.sda1.in‚0 disk.md0p1.out‚ 1265.02734375 disk.sdb2.out‚ 1312.7109375 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.rbd2.out‚ 3.3671875 proc.cpu.threads‚17ÓáKÎ4ß)/ _wandbruntime‚11939  _runtime‚11939¬ÃKéÉà)à trainer/global_step‚10099 % _timestamp‚1.7884758861790447e+09 _runtime‚11939.363980572 _step‚828 & lr-AdamW/pg1‚6.625489022718969e-05 & lr-AdamW/pg2‚6.625489022718969e-05OïÙ+Îá)È & lr-AdamW/pg1‚6.625489022718969e-05 & lr-AdamW/pg2‚6.625489022718969e-05 trainer/global_step‚10099 % _timestamp‚1.7884758861790447e+09 _runtime‚11939.363980572 _step‚828¼^Q2C4â)/ _wandbruntime‚11939  _runtime‚119396výSò'ã)ìO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚432037520694784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚456743980100096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20077695449600 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚432037269023232 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚411959807242240 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280133632 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20020772173858 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚436327520760320 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 467101765 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 114996648 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚456743676026368 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20416435399680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20416155266048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚411973051551986 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 114996646 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚436327544700416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚391952279378128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚432037269023232 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20077461780992 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚705 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0‹¯y÷'ä)ñO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚411973051551986 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 114996646 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚436327544700416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚391952279378128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚432037269023232 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20077461780992 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚705 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚432037520694784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚456743980100096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20077695449600 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚432037269023232 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚411959807242240 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280133632 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20020772173858 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚436327520760320 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 467101765 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 114996648 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚456743676026368 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20416435399680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20416155266048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792½R€,Bå)‚(:… îóçÔ˜®­‡ gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚134.908' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.727 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÕLþðæ)‚(:ô îóçÔŒ¦ÐŠ cpu‚25.200178422083354 network.sent‚ 2.7104027e+07 network.recv‚ 4.125832e+06 disk.rbd0.out‚ 3.3671875 disk.md0p1.out‚ 1266.8203125" proc.memory.rssMB‚ 3032.2265625 disk.rbd2.in‚0 disk.sdb.in‚5.40625 disk.sdb1.out‚0" disk.nvme0n1.out‚ 71479.4140625 disk.sda2.in‚ 5.56640625) proc.memory.availableMB‚ 9772.78515625) disk./.usagePercent‚83.17707412497406 disk.rbd0.in‚0.015625 disk.sda.out‚1314.568359375 disk.dm-0.in‚531405.14453125 proc.cpu.threads‚17% disk./.usageGB‚2974.8150596618652 disk.sdb2.out‚1314.568359375 disk.md0.out‚ 1266.8203125# disk.nvme0n1.in‚329983.51171875$ memory_percent‚40.35165309906006! disk.dm-0.out‚142639.39453125" disk.nvme1n1.in‚201421.6328125# disk.nvme1n1.out‚71159.98046875 disk.sdb2.in‚5.40625 disk.md0.in‚ 10.97265625 disk.rbd2.out‚ 3.3671875) proc.memory.percent‚18.50724220275879 disk.sdb.out‚1314.568359375 disk.sda.in‚ 5.56640625 disk.sda1.out‚0 disk.md0p1.in‚ 10.97265625 disk.sdb1.in‚0 disk.sda1.in‚0 disk.sda2.out‚1314.568359375Èúyþ4ç)/ _wandbruntime‚11940  _runtime‚11940ó€£#è)öM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20079659488768 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚432078770770944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 274812310 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 274785940 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20079425806848 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚73045882 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20418151446528 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 188052654 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 115007448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚391989809735552 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 192360794 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 192360792 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20022730663052 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚678 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 188053332 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 115007450 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚11940.501559858 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚456787666528768 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚436369204730368 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 467173104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚411999093279232 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚411999093279232 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20079659488768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 trainer/global_step‚10099 _step‚830 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 192360792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20022501681760 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚411999111282176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 192360794 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚73045206 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 467146732 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚432078519086080 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 274785940 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚436369228670464 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20418437858304 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceSta…MEòltsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20079425806848 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚432078770770944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 467173104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚411999111282176N”a‹'é)ûM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚73045882 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20418151446528 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 188052654 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 115007448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚391989809735552 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 192360794 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 192360792 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20022730663052 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚678 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 188053332 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 115007450 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚11940.501559858 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚456787666528768 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚436369204730368 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 467173104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚411999093279232 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚411999093279232 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20079659488768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 trainer/global_step‚10099 _step‚830 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 192360792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20022501681760 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚411999111282176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 192360794 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚73045206 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 467146732 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚432078519086080 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 274785940 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚436369228670464 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20418437858304 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20079425806848 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚432078770770944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 467173104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚411999111282176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20079659488768 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚432078770770944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 274812310 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 274785940 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20079425806848¾HHM4ê)/ _wandbruntime‚11940  _runtime‚119408¨Á\ë)Ö /loss/FRt4/assignment_loss‚3.250548569361369 /loss/FRt4/detection_loss‚3.1423003296057384 1loss/FBt2/assignment_loss‚0.17338274044921675 1loss/SRqqt1/detection_loss‚2.2220401473947473 .loss/FBt1/detection_loss‚2.526018667221069 trainer/global_step‚10099 2loss/SRqqt3/assignment_loss‚1.2223688012219212 2loss/SRqqt4/assignment_loss‚1.6256677745418115 0loss/SRqqt2/detection_loss‚2.071797560561787 $ _timestamp‚1.788475887317944e+09 _step‚831 0loss/FRt1/assignment_loss‚3.1997901984623502 0loss/FRt3/assignment_loss‚1.0781481629842287 1loss/FBt1/assignment_loss‚0.17452328981016194 .loss/FRt1/detection_loss‚2.909381854534149 1loss/SRqqt3/detection_loss‚0.8178384615045585 1loss/SRqqt4/detection_loss‚2.9837081987749445 epoch‚8 2loss/SRqqt2/assignment_loss‚1.2233136783946643 /loss/FBt3/detection_loss‚0.5091575141149488 1loss/SRqqt1/assignment_loss‚1.344996541537143 1loss/FBt3/assignment_loss‚0.07266137156454616 /loss/FRt3/detection_loss‚1.1570565372511938 .loss/FBt2/detection_loss‚2.273454889655113 &loss/total_loss‚41.232641742929374 _runtime‚11940.502815503 /loss/FRt2/assignment_loss‚3.028288029894537 /loss/FRt2/detection_loss‚2.0253529141025086 .loss/FBt4/detection_loss‚2.113355021266376 0loss/FBt4/assignment_loss‚0.0874904887202908ô˜ÀÑaì)Û 1loss/SRqqt4/detection_loss‚2.9837081987749445 epoch‚8 2loss/SRqqt2/assignment_loss‚1.2233136783946643 /loss/FBt3/detection_loss‚0.5091575141149488 1loss/SRqqt1/assignment_loss‚1.344996541537143 1loss/FBt3/assignment_loss‚0.07266137156454616 /loss/FRt3/detection_loss‚1.1570565372511938 .loss/FBt2/detection_loss‚2.273454889655113 &loss/total_loss‚41.232641742929374 _runtime‚11940.502815503 /loss/FRt2/assignment_loss‚3.028288029894537 /loss/FRt2/detection_loss‚2.0253529141025086 .loss/FBt4/detection_loss‚2.113355021266376 0loss/FBt4/assignment_loss‚0.0874904887202908 /loss/FRt4/assignment_loss‚3.250548569361369 /loss/FRt4/detection_loss‚3.1423003296057384 1loss/FBt2/assignment_loss‚0.17338274044921675 1loss/SRqqt1/detection_loss‚2.2220401473947473 .loss/FBt1/detection_loss‚2.526018667221069 trainer/global_step‚10099 2loss/SRqqt3/assignment_loss‚1.2223688012219212 2loss/SRqqt4/assignment_loss‚1.6256677745418115 0loss/SRqqt2/detection_loss‚2.071797560561787 $ _timestamp‚1.788475887317944e+09 _step‚831 0loss/FRt1/assignment_loss‚3.1997901984623502 0loss/FRt3/assignment_loss‚1.0781481629842287 1loss/FBt1/assignment_loss‚0.17452328981016194 .loss/FRt1/detection_loss‚2.909381854534149 1loss/SRqqt3/detection_loss‚0.8178384615045585¿Wôžñ‘í)‚(:† ýóçÔÖÄ€ˆ gpu.0.gpu‚37.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚101.986' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.4965 gpu.0.smClock‚1215 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì!¼V}î)‚(:ò ýóçÔÕ–ò‰ network.sent‚ 2.7200136e+07 network.recv‚ 4.132618e+06 disk.sda1.out‚0 disk.md0.out‚ 1268.171875 disk.sda.out‚1315.9892578125) disk./.usagePercent‚83.17596133564244$ disk./.usageGB‚2974.775260925293 disk.md0.in‚ 10.97265625# disk.nvme0n1.out‚71511.51953125 disk.md0p1.in‚ 10.97265625# disk.nvme1n1.out‚71160.79296875 disk.sdb.in‚5.40625$ memory_percent‚40.35050868988037 disk.sda2.in‚ 5.56640625 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 3.3671875) proc.memory.percent‚18.50724220275879 disk.md0p1.out‚ 1268.171875 proc.cpu.threads‚17 disk.rbd2.in‚0" disk.nvme1n1.in‚201421.6328125 disk.sdb1.in‚0 disk.sdb1.out‚0) proc.memory.availableMB‚ 9772.97265625 cpu‚25.199910372541357 disk.sda1.in‚0! disk.sda2.out‚1315.9892578125 disk.sdb2.in‚5.40625 disk.sdb.out‚1315.9892578125# disk.nvme0n1.in‚329983.51171875 disk.dm-0.in‚531405.14453125 disk.dm-0.out‚ 142672.3125" proc.memory.rssMB‚ 3032.2265625! disk.sdb2.out‚1315.9892578125 disk.rbd0.out‚ 3.3671875 disk.sda.in‚ 5.56640625ÇŠ*œï)‚(:‘ ŒôçÔ™Éé‡ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚119.978' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.994499999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ?!§ð)‚(:ö ŒôçÔ󉟈 disk.md0.in‚ 10.97265625 disk.md0.out‚ 1269.734375 network.sent‚ 2.7286804e+07% disk./.usageGB‚2974.8422355651855! disk.sdb.out‚1317.60888671875 disk.sdb1.in‚0$ memory_percent‚40.35220146179199 disk.sdb.in‚5.40625 disk.sda.in‚ 5.56640625! disk.dm-0.out‚142885.01171875 disk.sda2.in‚ 5.56640625 disk.md0p1.out‚ 1269.734375 cpu‚25.19966780823769 disk.sda1.in‚0 disk.sda1.out‚0" disk.sda2.out‚1317.60888671875 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 3.3671875( proc.memory.availableMB‚ 9772.6953125 network.recv‚ 4.139645e+06( disk./.usagePercent‚83.1778339746049# disk.nvme0n1.out‚71724.12109375 disk.dm-0.in‚531405.14453125 disk.md0p1.in‚ 10.97265625" disk.sdb2.out‚1317.60888671875 proc.cpu.threads‚17 disk.rbd2.in‚0" disk.nvme1n1.in‚201421.6328125! disk.sda.out‚1317.60888671875# proc.memory.rssMB‚ 3032.23046875( proc.memory.percent‚18.5072660446167 disk.rbd2.out‚ 3.3671875! disk.nvme1n1.out‚ 71160.890625# disk.nvme0n1.in‚329983.51171875 disk.sdb1.out‚0 disk.sdb2.in‚5.40625ÁM¢ñ)‚(:„ ›ôçÔµÉÔ‡ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚90.236' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚22.559 gpu.0.smClock‚1200 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñæ ~ò)‚(:ó ›ôçÔÄŸ‰‰ disk.rbd2.in‚0 disk.rbd2.out‚ 3.3671875 disk.sda1.out‚0 disk.sda2.in‚ 5.56640625( proc.memory.percent‚18.5072660446167 disk.sdb.out‚1319.2685546875 disk.sdb1.in‚0# disk.nvme0n1.in‚329983.51171875) disk./.usagePercent‚83.17870624443658 disk.sdb2.in‚5.40625 disk.md0.out‚ 1271.34375 disk.md0.in‚ 10.97265625 disk.rbd0.out‚ 3.3671875 disk.md0p1.in‚ 10.97265625 disk.rbd0.in‚0.015625 disk.sda.in‚ 5.56640625 disk.sda1.in‚0! disk.sda2.out‚1319.2763671875$ memory_percent‚40.35146236419678$ disk./.usageGB‚2974.873432159424" disk.nvme1n1.in‚201421.6328125 disk.dm-0.in‚531405.14453125 disk.md0p1.out‚ 1271.34375 network.recv‚ 4.143244e+06 disk.sdb1.out‚0 disk.sda.out‚1319.2763671875" disk.nvme0n1.out‚ 71798.4140625) proc.memory.availableMB‚ 9772.81640625# proc.memory.rssMB‚ 3032.23046875 cpu‚25.199414933220016 network.sent‚ 2.7292157e+07! disk.sdb2.out‚1319.2763671875# disk.nvme1n1.out‚71161.17578125 disk.sdb.in‚5.40625! disk.dm-0.out‚142959.58984375 proc.cpu.threads‚17úùq»’ó)‚(:‡ ªôçÔ÷éᇠgpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚136.743' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.18575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éå †ô)‚(:û ªôçÔˆÓü‡% disk./.usageGB‚2974.9673080444336 disk.md0p1.out‚ 1272.9921875 disk.md0.in‚ 10.97265625 disk.rbd2.out‚ 3.3671875) disk./.usagePercent‚83.18133105347158 disk.dm-0.in‚531405.14453125! disk.dm-0.out‚143154.54296875# disk.nvme1n1.out‚71161.27734375 disk.sdb2.in‚5.40625" disk.nvme1n1.in‚201421.6328125! disk.sda.out‚1320.97998046875( proc.memory.percent‚18.5072660446167 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd2.in‚0 cpu‚25.199171189737555 disk.sda2.in‚ 5.56640625" disk.sdb2.out‚1320.97998046875 disk.sdb.in‚5.40625! disk.sdb.out‚1320.97998046875 disk.sdb1.in‚0) proc.memory.availableMB‚ 9772.82421875 disk.rbd0.out‚ 3.3671875 disk.sdb1.out‚0 network.recv‚ 4.146957e+06! disk.nvme0n1.out‚ 71993.265625 disk.sda1.out‚0 disk.md0p1.in‚ 10.97265625 disk.sda.in‚ 5.56640625# disk.nvme0n1.in‚329983.51171875" disk.sda2.out‚1320.97998046875 disk.md0.out‚ 1272.9921875 disk.rbd0.in‚0.015625$ memory_percent‚40.35141468048096# proc.memory.rssMB‚ 3032.23046875 network.sent‚ 2.7295315e+07ãk!ª4õ)/ _wandbruntime‚12001  _runtime‚12001±Æ÷Éö)à & lr-AdamW/pg1‚6.305547885514218e-05 & lr-AdamW/pg2‚6.305547885514218e-05 trainer/global_step‚10149 % _timestamp‚1.7884759482963376e+09 _runtime‚12001.481507696 _step‚832X®ÂÎ÷)È _runtime‚12001.481507696 _step‚832 & lr-AdamW/pg1‚6.305547885514218e-05 & lr-AdamW/pg2‚6.305547885514218e-05 trainer/global_step‚10149 % _timestamp‚1.7884759482963376e+09ÀcÙD4ø)/ _wandbruntime‚12001  _runtime‚12001i$}øò'ù)ìO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 _runtime‚12001.483055133 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚413924127094784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 469379813 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚438411719262720 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 276123544 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 276097200 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 188920229 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 115536746 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 trainer/global_step‚10149 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20175911156224 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 193256269 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚434100038251008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20175677487616 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚458928106448896 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 188920915 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚413947507463694 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚434099786579456 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20516363246080 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚413924109091840 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚413924127094784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚438411743202816 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20118710214366 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚413947261443250 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚393828780209968 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20118481233282 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚73384167 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20516074723840 _step‚833 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 193256267 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20175911156224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚686 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 115536748 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚73383483 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚434099786579456 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 % _timestamp‚1.7884759482976382e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚413924109091840 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20175677487616 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 188920229 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 115536746 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 trainer/global_step‚10149 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20175911156224 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 193256269 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚434100? µÈ 038251008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20175677487616 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚458928106448896 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 188920915 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚413947507463694 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚434099786579456 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20516363246080 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚413924109091840 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚413924127094784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚438411743202816 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20118710214366 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚413947261443250 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚393828780209968 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20118481233282 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚73384167 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20516074723840 _step‚833 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 193256267 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20175911156224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚686 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 115536748 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚73383483 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚434099786579456 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 % _timestamp‚1.7884759482976382e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚413924109091840 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20175677487616 J°&ëü&ü)öM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚413963413131776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 _runtime‚12002.699571225 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 193274544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚438453427172864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 188938924 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 115547550 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚413963395128832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 469398434 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20518061656576 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 469398434 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20177875195392 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20518358554112 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚73391374 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 188938256 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚413986750289660 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 _step‚834 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚434141288327168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚666 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚413986996310312 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 193274542 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 469424806 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 % _timestamp‚1.7884759495125656e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20177641513472 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20177875195392 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 276150262 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚458971464889344 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 193274544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚668 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚438453403232768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚73390708 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 276123892 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 469424806 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 trainer/global_step‚10149 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 193274542 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚413963395128832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 115547548 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚393866310567392 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20177641513472 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 276123892 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚413963413131776þ—'ý)ûM _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚438453403232768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚73390708 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 276123892 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 469424806 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 trainer/global_step‚10149 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 193274542 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚413963395128832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 115547548 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚393866310567392 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20177641513472 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 276123892 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚413963413131776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚413963413131776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 _runtime‚12002.699571225 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 193274544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚438453427172864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 188938924 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 115547550 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚413963395128832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 469398434 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20518061656576 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 469398434 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20177875195392 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20518358554112 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚73391374 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 188938256 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚413986750289660 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 _step‚834 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚434141288327168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚666 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚413986996310312 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 193274542 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 469424806 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 % _timestamp‚1.7884759495125656e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20177641513472 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20177875195392 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 276150262 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚458971464889344 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 193274544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚668 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36“)F*4þ)/ _wandbruntime‚12002  _runtime‚12002^SÎcÿ)Ý /loss/FRt4/detection_loss‚3.2909862881615046 0loss/FRt1/assignment_loss‚2.3076744757849594 1loss/FBt1/assignment_loss‚0.21738679406677192 /loss/FBt1/detection_loss‚2.0616863548755644 &loss/total_loss‚38.559643798572665 2loss/SRqqt4/assignment_loss‚1.2480076998472214 0loss/FRt2/assignment_loss‚2.9117223864251915 2loss/SRqqt2/assignment_loss‚1.3232173780279775 1loss/FBt3/assignment_loss‚0.08306010301288852 1loss/SRqqt3/detection_loss‚0.7613183884136379 1loss/SRqqt4/detection_loss‚2.5333258626253707 /loss/FBt4/detection_loss‚2.6553302705287933 2loss/SRqqt1/assignment_loss‚1.5004206582521782 1loss/FBt2/assignment_loss‚0.11632044760818082 % _timestamp‚1.7884759495159774e+09 .loss/FRt1/detection_loss‚2.860989031092874 /loss/FRt2/detection_loss‚2.1077294609567616 0loss/SRqqt1/detection_loss‚1.955791005721459 /loss/FBt3/detection_loss‚0.5541597822015839 _step‚835 0loss/FRt3/assignment_loss‚1.1979732706513204 /loss/FRt4/assignment_loss‚2.651959090005784 1loss/SRqqt2/detection_loss‚2.1005619158667903 trainer/global_step‚10149 /loss/FBt2/detection_loss‚1.9549980163574219 epoch‚8 _runtime‚12002.700863761 2loss/SRqqt3/assignment_loss‚0.8052892516739667 1loss/FBt4/assignment_loss‚0.11923171607001255 /loss/FRt3/detection_loss‚1.2405041503444525øÉ©Ñh€*â .loss/FRt1/detection_loss‚2.860989031092874 /loss/FRt2/detection_loss‚2.1077294609567616 0loss/SRqqt1/detection_loss‚1.955791005721459 /loss/FBt3/detection_loss‚0.5541597822015839 _step‚835 0loss/FRt3/assignment_loss‚1.1979732706513204 /loss/FRt4/assignment_loss‚2.651959090005784 1loss/SRqqt2/detection_loss‚2.1005619158667903 trainer/global_step‚10149 /loss/FBt2/detection_loss‚1.9549980163574219 epoch‚8 _runtime‚12002.700863761 2loss/SRqqt3/assignment_loss‚0.8052892516739667 1loss/FBt4/assignment_loss‚0.11923171607001255 /loss/FRt3/detection_loss‚1.2405041503444525 /loss/FRt4/detection_loss‚3.2909862881615046 0loss/FRt1/assignment_loss‚2.3076744757849594 1loss/FBt1/assignment_loss‚0.21738679406677192 /loss/FBt1/detection_loss‚2.0616863548755644 &loss/total_loss‚38.559643798572665 2loss/SRqqt4/assignment_loss‚1.2480076998472214 0loss/FRt2/assignment_loss‚2.9117223864251915 2loss/SRqqt2/assignment_loss‚1.3232173780279775 1loss/FBt3/assignment_loss‚0.08306010301288852 1loss/SRqqt3/detection_loss‚0.7613183884136379 1loss/SRqqt4/detection_loss‚2.5333258626253707 /loss/FBt4/detection_loss‚2.6553302705287933 2loss/SRqqt1/assignment_loss‚1.5004206582521782 1loss/FBt2/assignment_loss‚0.11632044760818082 % _timestamp‚1.7884759495159774e+09Ã\95tx*‚(:í ¹ôçÔ쵑‰ disk.md0.out‚ 1274.42578125 cpu‚25.19895971585656 network.recv‚ 4.151353e+06 proc.cpu.threads‚17$ disk./.usageGB‚2975.092529296875 disk.sdb1.out‚0 disk.nvme0n1.out‚ 72095.21875 disk.dm-0.in‚531405.14453125 disk.md0.in‚ 10.97265625" disk.nvme1n1.out‚ 71161.4921875 disk.sda.in‚ 5.56640625# disk.nvme0n1.in‚329983.51171875 disk.sdb2.out‚1322.482421875 disk.rbd0.out‚ 3.3671875( proc.memory.availableMB‚ 9773.0546875 network.sent‚ 2.7301175e+07 disk.sdb.out‚1322.482421875 disk.sda1.in‚0 disk.sda2.out‚1322.482421875 disk.sdb2.in‚5.40625# proc.memory.rssMB‚ 3032.23046875' disk./.usagePercent‚83.184832292099 disk.sdb1.in‚0 disk.dm-0.out‚143256.7109375 disk.sda1.out‚0 disk.sda2.in‚ 5.56640625 disk.rbd2.in‚0" disk.nvme1n1.in‚201421.6328125 disk.sdb.in‚5.40625 disk.sda.out‚1322.482421875 disk.md0p1.in‚ 10.97265625 disk.rbd2.out‚ 3.3671875( proc.memory.percent‚18.5072660446167 disk.md0p1.out‚ 1274.42578125 disk.rbd0.in‚0.015625$ memory_percent‚40.35000801086426«Þ˜‘‚*‚(:† ¹ôçÔÈ ’‰ gpu.0.gpu‚39.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚99.201' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚24.80025 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p ifƒ*‚(:… ÈôçÔ÷•½‡ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.9487457 Ÿ]Ã*27539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚131.79' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.9475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04ZAtƒ„*‚(:ø ÈôçÔŸ¼§ˆ disk.sda1.in‚0 disk.md0p1.out‚ 1275.80859375 disk.sdb2.in‚5.40625$ memory_percent‚40.35193920135498% disk./.usageGB‚2975.0921325683594 disk.rbd2.out‚ 3.3671875! disk.sdb.out‚1323.92724609375 disk.sda2.in‚ 5.56640625 disk.md0p1.in‚ 10.97265625 disk.md0.in‚ 10.97265625 disk.sdb1.in‚0" disk.nvme1n1.in‚201421.6328125 disk.sdb1.out‚0 disk.sda.in‚ 5.56640625" disk.sda2.out‚1323.92724609375( proc.memory.percent‚18.5072660446167 cpu‚25.19873657226496 network.sent‚ 2.7408888e+07 disk.rbd0.in‚0.015625" disk.sdb2.out‚1323.92724609375) proc.memory.availableMB‚ 9772.73828125 network.recv‚ 4.158456e+06) disk./.usagePercent‚83.18482119940366 disk.rbd2.in‚0# disk.nvme1n1.out‚71161.61328125# disk.nvme0n1.in‚329983.51171875# proc.memory.rssMB‚ 3032.23046875 disk.md0.out‚ 1275.80859375 disk.rbd0.out‚ 3.3671875! disk.sda.out‚1323.92724609375# disk.nvme0n1.out‚72406.13671875 disk.dm-0.out‚ 143567.75 disk.sdb.in‚5.40625 disk.dm-0.in‚531405.14453125 disk.sda1.out‚0 proc.cpu.threads‚17’¡±…*‚(:„ ×ôçÔþï㇠gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚89.904' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚22.476 gpu.0.smClock‚1200 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uð›|†*‚(:ñ ×ôçÔú”Ɉ disk.sda2.out‚1326.134765625 disk.rbd2.out‚ 3.3671875 disk.sdb1.out‚0 proc.cpu.threads‚17 disk.sdb.in‚5.40625$ memory_percent‚40.35212993621826 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 3.3671875 disk.rbd2.in‚0 disk.md0p1.out‚ 1277.953125 network.recv‚ 4.162161e+06) disk./.usagePercent‚83.18483239875954# disk.nvme0n1.in‚329983.51171875 disk.dm-0.in‚531405.14453125% disk./.usageGB‚2975.0925331115723 disk.sda1.out‚0 disk.md0p1.in‚ 10.97265625 disk.sdb2.in‚5.40625 disk.md0.out‚ 1277.953125 disk.sda.out‚1326.134765625) proc.memory.availableMB‚ 9772.70703125# proc.memory.rssMB‚ 3032.23046875! disk.nvme0n1.out‚ 72558.546875! disk.dm-0.out‚143720.39453125 disk.sda1.in‚0 disk.sdb2.out‚1326.134765625 disk.sdb.out‚1326.134765625 disk.md0.in‚ 10.97265625" disk.nvme1n1.in‚201421.6328125( proc.memory.percent‚18.5072660446167 disk.sda2.in‚ 5.56640625 disk.sda.in‚ 5.56640625 cpu‚25.198530020877055 network.sent‚ 2.7416426e+07# disk.nvme1n1.out‚71161.84765625 disk.sdb1.in‚0þ¾3k‡*‚(:… æôçÔ۴݇ gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚91.186' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚22.7965 gpu.0.smClock‚1215 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-ˆfpˆ*‚(:å æôçÔÂ阈% disk./.usageGB‚2975.0925331115723 disk.sda.in‚ 5.56640625" disk.nvme0n1.out‚ 72684.2890625 disk.sda2.in‚ 5.56640625 disk.md0p1.in‚ 10.97265625 disk.rbd0.in‚0.015625# proc.memory.rssMB‚ 3032.23046875 disk.sda.out‚1327.7451171875 disk.dm-0.in‚531405.14453125! disk.nvme1n1.out‚ 71161.953125 cpu‚25.19832676556886) disk./.usagePercent‚83.18483239875954 disk.dm-0.out‚143846.2421875( proc.memory.percent‚18.5072660446167 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0" disk.nvme1n1.in‚201421.6328125 disk.sdb.in‚5.40625$ memory_percent‚40.35358428955078 disk.rbd0.out‚ 3.3671875 disk.rbd2.in‚0 disk.rbd2.out‚ 3.3671875 network.sent‚ 2.7422717e+07 network.recv‚ 4.166477e+06! disk.sda2.out‚1327.7451171875 disk.md0.in‚ 10.97265625 disk.md0.out‚1279.5 disk.md0p1.out‚1279.5 disk.sdb2.in‚5.40625! disk.sdb2.out‚1327.7451171875# disk.nvme0n1.in‚329983.51171875 disk.sdb.out‚1327.7451171875& proc.memory.availableMB‚ 9772.46875 disk.sda1.out‚0žõÃ4‰*/ _wandbruntime‚12063  _runtime‚12063#¨{µËŠ*Å ' lr-AdamW/pg1‚5.9928834169514746e-05 ' lr-AdamW/pg2‚5.9928834169514746e-05 trainer/global_step‚10199 % _timestamp‚1.7884760104203386e+09 _runtime‚12063.605400123 _step‚836ȯåNЋ*Ê _runtime‚12063.605400123 _step‚836 ' lr-AdamW/pg1‚5.9928834169514746e-05 ' lr-AdamW/pg2‚5.9928834169514746e-05 trainer/global_step‚10199 % _timestamp‚1.7884760104203386e+09ÄI¯Ò—4Œ*/ _wandbruntime‚12063  _runtime‚12063qöÀ°ñ'*ëO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20615942856192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20273893194240 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚73736664 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 277461496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚395705315120528 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 471631515 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚461111860621312 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20616227184128 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚415888410941440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20274126862848 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚436162555807232 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 189812822 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 194170019 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚440495917765120 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 116076848 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚461112168889344 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚10199 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚73735976 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚440495941705216 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚415921963375402 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 194170017 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚415921717354958 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 471605169 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚415888428944384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 _runtime‚12063.606539638 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚415888410941440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚395705298081168 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 277435152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚436162555807232 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 116076846 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚436162304135680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20216648254874 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚436162304135680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 277435152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚436162555807232 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 116076846 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚436162304135680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20216648254874 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚436162304135680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 189812822 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 194170019 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚440495917765120 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 116076848 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚461112168889344 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚10199 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚73735976 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚440495941705216 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚415921963375402 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 194170017 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚415921717354958 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 471605169 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚415888428944384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 _runtime‚12063.606539638 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚415888410941440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344ÅF®cå4*/ _wandbruntime‚12064  _runtime‚12064g jÈ*öM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚436203554198528 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20617931223552 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 194188294 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚12064.776624028 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 189830419 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 277461844 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚395742845477952 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 471650136 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚691 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 116087648 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 277488214 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 116087650 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 471650136 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 HBDeviceStatsMonitor.on_train_batch_end/oversize_aÚ Åÿ4 llocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 % _timestamp‚1.7884760115897355e+09 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚415961206201368 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚415927714981376 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚73743462 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚440537625675264 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20618228121088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20218377762776 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚395742828438592 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 _step‚838 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚436203805883392 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 277488214 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20275857220096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚73742771 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 471676508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚693 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚461155532958720)ÔEH'‘*ûM WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 116087650 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 471650136 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 % _timestamp‚1.7884760115897355e+09 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚415961206201368 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚415927714981376 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚73743462 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚440537625675264 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20618228121088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20218377762776 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚395742828438592 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 _step‚838 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚436203805883392 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 277488214 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20275857220096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚73742771 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 471676508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚693 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚461155532958720 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚436203554198528 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20617931223552 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 194188294 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚12064.776624028 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 189830419 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 277461844 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚395742845477952 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 471650136 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚691 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 116087648 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 277488214 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0Æ=òC4’*/ _wandbruntime‚12064  _runtime‚12064hþBÌ_“*Ù 0loss/FRt2/assignment_loss‚3.4095650661376213 .loss/FRt2/detection_loss‚2.658907210493776 /loss/FRt3/detection_loss‚1.2114616259725557 % _timestamp‚1.7884760115933404e+09 _step‚839 2loss/SRqqt1/assignment_loss‚1.3057556915524844 0loss/SRqqt1/detection_loss‚2.186436423578778 /loss/FBt3/detection_loss‚0.5225669018188377 &loss/total_loss‚41.014573154677365 epoch‚8 1loss/FBt1/assignment_loss‚0.09817220066159038 1loss/FBt2/assignment_loss‚0.14371665754431062 /loss/FRt1/detection_loss‚2.8016628661463336 0loss/SRqqt2/detection_loss‚2.397106624418689 trainer/global_step‚10199 0loss/FRt4/assignment_loss‚3.5508741537729898 1loss/FBt3/assignment_loss‚0.05697451947412683 1loss/SRqqt3/detection_loss‚0.9474415139993653 1loss/SRqqt4/detection_loss‚2.2146336273713545 /loss/FBt2/detection_loss‚2.6743954924436717 1loss/SRqqt4/assignment_loss‚1.387373970765056 1loss/FBt4/assignment_loss‚0.10720642645473495 .loss/FRt4/detection_loss‚3.167083842886819 .loss/FBt1/detection_loss‚2.330024905502796 0loss/FRt3/assignment_loss‚1.1258702165550656 _runtime‚12064.778188855 2loss/SRqqt2/assignment_loss‚1.1259966261925236 2loss/SRqqt3/assignment_loss‚0.7760142305802825 .loss/FBt4/detection_loss‚2.336399281726164 0loss/FRt1/assignment_loss‚2.4789330786274326¢5Ód”*Þ /loss/FBt2/detection_loss‚2.6743954924436717 1loss/SRqqt4/assignment_loss‚1.387373970765056 1loss/FBt4/assignment_loss‚0.10720642645473495 .loss/FRt4/detection_loss‚3.167083842886819 .loss/FBt1/detection_loss‚2.330024905502796 0loss/FRt3/assignment_loss‚1.1258702165550656 _runtime‚12064.778188855 2loss/SRqqt2/assignment_loss‚1.1259966261925236 2loss/SRqqt3/assignment_loss‚0.7760142305802825 .loss/FBt4/detection_loss‚2.336399281726164 0loss/FRt1/assignment_loss‚2.4789330786274326 0loss/FRt2/assignment_loss‚3.4095650661376213 .loss/FRt2/detection_loss‚2.658907210493776 /loss/FRt3/detection_loss‚1.2114616259725557 % _timestamp‚1.7884760115933404e+09 _step‚839 2loss/SRqqt1/assignment_loss‚1.3057556915524844 0loss/SRqqt1/detection_loss‚2.186436423578778 /loss/FBt3/detection_loss‚0.5225669018188377 &loss/total_loss‚41.014573154677365 epoch‚8 1loss/FBt1/assignment_loss‚0.09817220066159038 1loss/FBt2/assignment_loss‚0.14371665754431062 /loss/FRt1/detection_loss‚2.8016628661463336 0loss/SRqqt2/detection_loss‚2.397106624418689 trainer/global_step‚10199 0loss/FRt4/assignment_loss‚3.5508741537729898 1loss/FBt3/assignment_loss‚0.05697451947412683 1loss/SRqqt3/detection_loss‚0.9474415139993653 1loss/SRqqt4/detection_loss‚2.2146336273713545ÇPsʤ•*‚(:… õôçÔ¥Ýæ† gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚130.564' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.641 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³íw–*‚(:ì õôçÔÁŽíˆ" disk.sdb2.out‚1329.43994140625 disk.sda.in‚ 5.56640625 disk.sda1.in‚0 cpu‚25.198098857769505 network.sent‚ 2.7439364e+07 network.recv‚ 4.170794e+06 disk.dm-0.in‚531405.14453125 disk.md0p1.in‚ 10.97265625 proc.cpu.threads‚17$ disk./.usageGB‚2974.817844390869 disk.rbd2.out‚ 3.3671875! disk.sda.out‚1329.43994140625" disk.sda2.out‚1329.43994140625 disk.md0.out‚1281.125 disk.sdb1.out‚0 disk.nvme0n1.out‚ 72852.625 disk.dm-0.out‚144015.0703125( proc.memory.availableMB‚ 9772.8203125( proc.memory.percent‚18.5072660446167) disk./.usagePercent‚83.17715198716252 disk.sdb2.in‚5.40625 disk.rbd2.in‚0" disk.nvme1n1.in‚201421.6328125 disk.sda2.in‚ 5.56640625$ memory_percent‚40.35143852233887 disk.md0.in‚ 10.97265625" disk.nvme1n1.out‚ 71162.4453125# disk.nvme0n1.in‚329983.51171875 disk.rbd0.in‚0.015625! disk.sdb.out‚1329.43994140625 disk.sdb1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 3032.23046875 disk.rbd0.out‚ 3.3671875 disk.sdb.in‚5.40625 disk.md0p1.out‚1281.125a¶õF—*‚(:… „õçÔÛȇ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚139.528' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 —:"ƒ˜*‚(:ø „õçÔÔ¤Œˆ( proc.memory.percent‚18.5072660446167 disk.rbd2.in‚0! disk.sdb.out‚1331.37646484375 network.sent‚ 2.7536458e+07 network.recv‚ 4.177252e+06# disk.nvme0n1.out‚73039.93359375! disk.dm-0.out‚144202.51171875 disk.sdb1.in‚0$ memory_percent‚40.34996032714844% disk./.usageGB‚2974.5927543640137% proc.memory.availableMB‚ 9773.0625 proc.cpu.threads‚17) disk./.usagePercent‚83.17085837580413 disk.dm-0.in‚531405.14453125 disk.sdb2.in‚5.40625 disk.md0.out‚ 1283.00390625 disk.rbd0.out‚ 3.3671875 disk.sdb1.out‚0 disk.md0p1.out‚ 1283.00390625 cpu‚25.19791910597979 disk.sda1.out‚0 disk.md0p1.in‚ 10.97265625" disk.sdb2.out‚1331.37646484375 disk.rbd0.in‚0.015625! disk.sda.out‚1331.37646484375 disk.sda2.in‚ 5.56640625" disk.nvme1n1.in‚201421.6328125 disk.sdb.in‚5.40625 disk.sda.in‚ 5.56640625 disk.sda1.in‚0 disk.md0.in‚ 10.97265625 disk.rbd2.out‚ 3.3671875# proc.memory.rssMB‚ 3032.23046875# disk.nvme0n1.in‚329983.51171875" disk.sda2.out‚1331.37646484375! disk.nvme1n1.out‚ 71162.578125Ì”õÞ‘™*‚(:† “õçÔ½±ú‡ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚140.754' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.1885 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷û©ƒš*‚(:ø “õçÔ†–±Š disk.sdb.in‚5.40625! disk.sdb.out‚1332.84130859375! disk.sda.out‚1332.84130859375 disk.md0p1.in‚ 10.97265625# proc.memory.rssMB‚ 3032.23046875( proc.memory.percent‚18.5072660446167" disk.sdb2.out‚1332.84130859375 disk.md0.in‚ 10.97265625 disk.sdb1.in‚0 disk.md0.out‚ 1284.41015625 disk.rbd0.out‚ 3.3671875 proc.cpu.threads‚17$ disk./.usageGB‚2973.778839111328# disk.nvme1n1.out‚71162.73046875 disk.sda.in‚ 5.56640625# disk.nvme0n1.out‚73229.92578125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚ 5.56640625# disk.nvme0n1.in‚329983.51171875 disk.dm-0.in‚531405.14453125$ memory_percent‚40.34934043884277 network.recv‚ 4.181091e+06) disk./.usagePercent‚83.14810096468905 disk.rbd2.out‚ 3.3671875" disk.sda2.out‚1332.84130859375 network.sent‚ 2.7541669e+07 disk.rbd0.in‚0.015625" disk.nvme1n1.in‚201421.6328125 disk.dm-0.out‚ 144392.65625 disk.sdb2.in‚5.40625 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.md0p1.out‚ 1284.41015625( proc.memory.availableMB‚ 9773.1640625 cpu‚25.1976978487249N–°<4›*/ _wandbruntime‚12117  _runtime‚12117" Ëœ*Å ' lr-AdamW/pg1‚5.6875607758373704e-05 ' lr-AdamW/pg2‚5.6875607758373704e-05 trainer/global_step‚10249 % _timestamp‚1.7884760644177415e+09 _runtime‚12117.602692042 _step‚840[ï²JÐ*Ê ' lr-AdamW/pg2‚5.6875607758373704e-05 trainer/global_step‚10249 % _timestamp‚1.7884760644177415e+09 _runtime‚12117.602692042 _step‚840 ' lr-AdamW/pg1‚5.6875607758373704e-05ȧ—h4ž*/ _wandbruntime‚12117  _runtime‚12117…–Sñ'Ÿ*ëO JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 195083769 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 278773104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚841 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20372108900864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 190700285 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20715790961152 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚686 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20372342569472 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 473883217 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚417852712791040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚74083339 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚397581815952368 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 trainer/global_step‚10249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚463296234371072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 195083767 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚438224821691904 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 278773104 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 327142400 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚438225073363456 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚442580140207616 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚442580116267520 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20314586295382 _runtime‚12117.603764498 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 278799448 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚417852730793984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20372108900864 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 116616946 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 $ _timestamp‚1.788476064418761e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 278799448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚397581832991728 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚417852712791040 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20716094163456 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 473856871 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20314357314298 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚438224821691904 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚463295907228672 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚417896419287110 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552ОøÆ; *ðO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚397581832991728 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚417852712791040 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20716094163456 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 473856871 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20314357314298 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚438224821691904 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚463295907228672 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚417896419287110 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QIDevic¢¾;Ô»!eStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 195083769 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 278773104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚841 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20372108900864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 190700285 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20715790961152 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚686 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20372342569472 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 473883217 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚417852712791040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚74083339 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚397581815952368 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 trainer/global_step‚10249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚463296234371072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 195083767 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚438224821691904 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 278773104 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 327142400 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚438225073363456 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚442580140207616 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚442580116267520 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20314586295382 _runtime‚12117.603764498 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 278799448 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚417852730793984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20372108900864 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 116616946 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 $ _timestamp‚1.788476064418761e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 278799448ÉFdHv4¡*/ _wandbruntime‚12118  _runtime‚12118)=–„ü&¢*öM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 473928210 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 278826166 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 278799796 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 trainer/global_step‚10249 % _timestamp‚1.7884760653543184e+09 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 116627748 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚74090639 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚842 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 195102044 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚442621824177664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 116627750 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚417935908133728 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 473901838 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 195102042 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 473928210 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 473901838 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚417891998828032 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚463339583296000 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20316544784576 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 190717703 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309®Çì|'£*ûM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 190718389 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚397619346309792 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚438266323439616 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20374306608640 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 195102044 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚463339902036480 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20316315803284 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 190717703 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 473928210 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 278826166 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 278799796 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 trainer/global_step‚10249 % _timestamp‚1.7884760653543184e+09 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 116627748 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚74090639 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚842 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 195102044 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚442621824177664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 116627750 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚417935908133728 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 473901838 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 195102042 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 473928210 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 473901838 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚417891998828032 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚463339583296000 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20316544784576 PÇ<ªñ'³*ëO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚419870875198818 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚440287339248128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚440287590919680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 $ _timestamp‚1.788476118064381e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 280137400 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 195997519 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 195997517 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚419817032643584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20412295354806 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚444664338710016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 _runtime‚12171.249399293 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 195997519 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚419817014640640 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚444664314769920 trainer/global_step‚10299 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚440287590919680 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚419817014640640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 _step‚845 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 117157048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚74432138 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20470558276096 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚74432812 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚440287339248128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚419817032643584 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 280137400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 191589184 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚465479932682240 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20470324607488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 280111056 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 191589860 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚676 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚465480245144576 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20470558276096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20815906434560 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚399458350862928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 280111056 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 117157046 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 $ _timestamp‚1.788476118064381e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 280137400 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 195997519 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 195997517 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚419817032643584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20412295354806 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚444664338710016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 _runtime‚12171.249399293 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 195997519 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚419817014640640 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚444664314769920 trainer/global_step‚10299 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚440287590919680 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚419817014640640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 _step‚845 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 117157048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚74432138 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20470558276096 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚74432812 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚440287339248128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚419817032643584 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 280137400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 191589184 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚465479932682240 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20470324607488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 280111056 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 191589860 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚676 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚465480245144576 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20470558276096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20815906434560 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚399458350862928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 280111056 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 117157046 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608Í`»íø4µ*/ _wandbruntime‚12172  _runtime‚12172räoȶ*öM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚654 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚440328840995840 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚419856318680576 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 280137748 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 476153540 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚419910118024784 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 280164118 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 196015792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚74439091 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚465523619053056 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 196015794 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 280137748 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 117167848 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20414253843792 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚419856318680576 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 280164118 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚419856300677632 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 191607595 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚465523925210624 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 117167850 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚12172.183805769 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 476179912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20472288633344 L>DeviceStatsMonitor.on_g’ðFÞ train_batch_end/allocation.all.allocated‚ 476179912 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚399495864180992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 476153540 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20472522315264 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20817620313088 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚419910364045436 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚656 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20414482825084 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚74439745 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20817902530560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 191606939 trainer/global_step‚10299 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚440328589310976 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 196015794 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0nnOæ'·*ûM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20472522315264 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20817620313088 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚419910364045436 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚656 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20414482825084 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚74439745 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20817902530560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 191606939 trainer/global_step‚10299 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚440328589310976 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 196015794 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚654 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚440328840995840 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚419856318680576 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 280137748 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 476153540 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚419910118024784 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 280164118 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 196015792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚74439091 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚465523619053056 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 196015794 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 280137748 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 117167848 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20414253843792 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚419856318680576 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 280164118 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚419856300677632 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 191607595 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚465523925210624 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 117167850 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚12172.183805769 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 476179912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20472288633344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 476179912 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚399495864180992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 476153540 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249Îà ˆK4¸*/ _wandbruntime‚12172  _runtime‚12172‚\¹*Ö 1loss/SRqqt1/assignment_loss‚1.190359537090574 0loss/FBt1/assignment_loss‚0.1269502137755167 1loss/SRqqt2/assignment_loss‚1.233126415638253 /loss/FRt1/detection_loss‚2.8928756867685625 1loss/SRqqt3/detection_loss‚0.6983067723648513 .loss/FBt1/detection_loss‚2.403034741228277 0loss/FRt4/assignment_loss‚3.5619491219520567 1loss/FBt2/assignment_loss‚0.22532825267504897 epoch‚8 trainer/global_step‚10299 0loss/FRt1/assignment_loss‚2.9060983330972734 0loss/FRt3/assignment_loss‚1.0966319747873254 1loss/FBt4/assignment_loss‚0.05446111616522901 .loss/FBt2/detection_loss‚2.474564883444044 .loss/FRt2/detection_loss‚2.015448332797839 /loss/FRt3/detection_loss‚1.1956745335781895 0loss/SRqqt1/detection_loss‚2.248680342095239 .loss/FBt3/detection_loss‚0.474028398525225 0loss/FRt2/assignment_loss‚3.5458341664738127 1loss/FBt3/assignment_loss‚0.11367301245762171 .loss/FBt4/detection_loss‚2.342800180117289 _runtime‚12172.185771108 2loss/SRqqt3/assignment_loss‚0.7271099383121028 2loss/SRqqt4/assignment_loss‚1.9436494326218963 1loss/SRqqt4/detection_loss‚2.7135529232521853 %loss/total_loss‚41.32070285682423 /loss/FRt4/detection_loss‚3.1104691004753113 1loss/SRqqt2/detection_loss‚2.0260954471305013 % _timestamp‚1.7884761190010743e+09 _step‚847Îu°aº*Û .loss/FRt2/detection_loss‚2.015448332797839 /loss/FRt3/detection_loss‚1.1956745335781895 0loss/SRqqt1/detection_loss‚2.248680342095239 .loss/FBt3/detection_loss‚0.474028398525225 0loss/FRt2/assignment_loss‚3.5458341664738127 1loss/FBt3/assignment_loss‚0.11367301245762171 .loss/FBt4/detection_loss‚2.342800180117289 _runtime‚12172.185771108 2loss/SRqqt3/assignment_loss‚0.7271099383121028 2loss/SRqqt4/assignment_loss‚1.9436494326218963 1loss/SRqqt4/detection_loss‚2.7135529232521853 %loss/total_loss‚41.32070285682423 /loss/FRt4/detection_loss‚3.1104691004753113 1loss/SRqqt2/detection_loss‚2.0260954471305013 % _timestamp‚1.7884761190010743e+09 _step‚847 1loss/SRqqt1/assignment_loss‚1.190359537090574 0loss/FBt1/assignment_loss‚0.1269502137755167 1loss/SRqqt2/assignment_loss‚1.233126415638253 /loss/FRt1/detection_loss‚2.8928756867685625 1loss/SRqqt3/detection_loss‚0.6983067723648513 .loss/FBt1/detection_loss‚2.403034741228277 0loss/FRt4/assignment_loss‚3.5619491219520567 1loss/FBt2/assignment_loss‚0.22532825267504897 epoch‚8 trainer/global_step‚10299 0loss/FRt1/assignment_loss‚2.9060983330972734 0loss/FRt3/assignment_loss‚1.0966319747873254 1loss/FBt4/assignment_loss‚0.05446111616522901 .loss/FBt2/detection_loss‚2.474564883444044ÏQolœ»*‚(:‘ ÞõçÔ·éΆ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z^3„|¼*‚(:ñ ÞõçÔóù† cpu‚25.196629148402245 disk.dm-0.out‚144728.5546875 disk.md0p1.out‚ 1292.3515625 network.recv‚ 4.205498e+06% disk./.usageGB‚2973.9057998657227 disk.rbd2.in‚0# disk.nvme1n1.out‚71163.74609375# proc.memory.rssMB‚ 3032.23046875 proc.cpu.threads‚17 disk.rbd0.in‚0.015625 disk.sda.out‚ 1341.08984375) proc.memory.availableMB‚ 9772.33984375 disk.md0p1.in‚ 10.97265625 disk.md0.in‚ 10.97265625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb2.out‚ 1341.08984375% memory_percent‚40.354371070861816 network.sent‚ 2.7671085e+07 disk.sdb.in‚5.40625) disk./.usagePercent‚83.15165084051912" disk.nvme1n1.in‚201421.6328125 disk.sdb1.out‚0# disk.nvme0n1.in‚329983.51171875# disk.nvme0n1.out‚73564.80859375 disk.rbd0.out‚ 3.3671875 disk.rbd2.out‚ 3.3671875 disk.sda.in‚ 5.56640625 disk.sda2.in‚ 5.56640625 disk.sdb2.in‚5.40625 disk.sdb.out‚ 1341.08984375 disk.md0.out‚ 1292.3515625 disk.sdb1.in‚0 disk.dm-0.in‚531405.14453125 disk.sda2.out‚ 1341.08984375( proc.memory.percent‚18.5072660446167D`!Û‘½*‚(:† íõçÔ…Ý÷… gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚132.122' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.0305 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08 È£x¾*‚(:í íõçÔóÑ¥ˆ disk.rbd2.in‚0 disk.sda.out‚1342.8994140625( proc.memory.availableMB‚ 9772.3203125 cpu‚25.19642305979493 proc.cpu.threads‚17 disk.md0.out‚ 1294.09375$ memory_percent‚40.35449028015137( proc.memory.percent‚18.5072660446167 disk.rbd2.out‚ 3.3671875 disk.nvme1n1.out‚ 71164.125 disk.md0p1.out‚ 1294.09375 disk.dm-0.in‚531405.14453125) disk./.usagePercent‚83.15164934727167 disk.rbd0.out‚ 3.3671875 disk.sdb.in‚5.40625 disk.sdb1.out‚0# disk.nvme0n1.in‚329983.51171875 disk.sdb2.in‚5.40625" disk.nvme1n1.in‚201421.6328125 disk.sdb.out‚1342.8994140625! disk.dm-0.out‚144882.87890625 disk.sda1.in‚0 network.sent‚ 2.7781047e+07 network.recv‚ 4.212907e+06 disk.sdb1.in‚0 disk.md0p1.in‚ 10.97265625# proc.memory.rssMB‚ 3032.23046875 disk.sda2.in‚ 5.56640625! disk.sda2.out‚1342.8994140625$ disk./.usageGB‚2973.905746459961! disk.sdb2.out‚1342.8994140625 disk.md0.in‚ 10.97265625 disk.sda.in‚ 5.56640625# disk.nvme0n1.out‚73728.75390625 disk.sda1.out‚0 disk.rbd0.in‚0.015625*mÎù’¿*‚(:‡ üõçÔªñ… gpu.0.gpu‚23.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚119.701' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.92525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€?Ù¤À*‚(:ö üõçÔøÃö‡ disk.md0.out‚ 1295.54296875 disk.rbd2.in‚0" disk.nvme1n1.in‚201421.6328125 disk.sdb1.out‚0 disk.md0p1.in‚ 10.97265625 network.sent‚ 2.7786889e+07 disk.sdb.in‚5.40625# disk.nvme0n1.out‚73785.88671875 disk.sda2.in‚ 5.56640625( disk./.usagePercent‚83.1439650958955% disk./.usageGB‚2973.6309204101562! disk.sdb2.out‚1344.4072265625 disk.sda.in‚ 5.56640625 disk.md0p1.out‚ 1295.54296875 proc.cpu.threads‚17 network.recv‚ 4.216646e+06! disk.nvme1n1.out‚ 71164.296875 disk.dm-0.in‚531405.14453125! disk.sda2.out‚1344.4072265625 disk.sdb1.in‚0# disk.nvme0n1.in‚329983.51171875 disk.sda1.out‚0' proc.memory.availableMB‚ 9772.828125 cpu‚25.196204399688007 disk.rbd0.out‚ 3.3671875 disk.rbd2.out‚ 3.3671875 disk.sdb.out‚1344.4072265625 disk.sda1.in‚0 disk.rbd0.in‚0.015625 disk.sda.out‚1344.4072265625! disk.dm-0.out‚144950.18359375$ memory_percent‚40.35139083862305# proc.memory.rssMB‚ 3032.23046875( proc.memory.percent‚18.5072660446167 disk.sdb2.in‚5.40625 disk.md0.in‚ 10.97265625ÿ˜¯G4Á*/ _wandbruntime‚12220  _runtime‚12220d¿WÔËÂ*Å % _timestamp‚1.7884761674596696e+09 _runtime‚12220.644637199 _step‚848 ' lr-AdamW/pg1‚5.0991939477922364e-05 ' lr-AdamW/pg2‚5.0991939477922364e-05 trainer/global_step‚10349ò õÐÃ*Ê ' lr-AdamW/pg1‚5.0991939477922364e-05 ' lr-AdamW/pg2‚5.0991939477922364e-05 trainer/global_step‚10349 % _timestamp‚1.7884761674596696e+09 _runtime‚12220.644637199 _step‚848Т©ˆ4Ä*/ _wandbruntime‚12220  _runtime‚12220SueÍò'Å*ìO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20568773982720 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20568540314112 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20510233395314 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚467664276248576 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚421845331110526 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 117697148 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚421845085090082 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 192480409 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 281449008 _step‚849 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚421781334493184 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 117697146 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚446748537212416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚442349856804352 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 % _timestamp‚1.7884761674606805e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20568773982720 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚12220.645692774 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 478386621 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 281475352 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 281475352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚74783263 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚401334851694768 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20568540314112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 196911269 trainer/global_step‚10349 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚442349856804352 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 192481087 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚467663967980544 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚442350108475904 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚678 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚401334868734128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚421781316490240 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚442350108475904 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 196911269 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚442349856804352 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 % _timestamp‚1.7884761674606805e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20568773982720 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚12220.645692774 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 478386621 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 281475352 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 281475352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚74783263 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚401334851694768 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20568540314112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 196911269 trainer/global_step‚10349 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚442349856804352 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 192481087 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚467663967980544 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚442350108475904 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚678 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚401334868734128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚421781316490240 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚442350108475904 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 196911269 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚467664276248576 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚421845331110526 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 117697148 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚421845085090082 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 192480409 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 281449008 _step‚849 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚421781334493184 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736Ñ^g~4Ç*/ _wandbruntime‚12221  _runtime‚12221Ýï…øü&È*öM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 281502070 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚421820620530176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 192497864 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20570738021888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚74790593 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 478405242 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 117707950 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 117707948 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 196929544 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884761683938806e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20570738021888 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚467707645495296 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20917448252928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚401372382052192 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20512191884300 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 196929544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚442391358552064 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚401372399091552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚446790197242368 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20570504339968 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚677 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 281502070 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 478431614 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36*"Ê'É*ûM UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20570738021888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚74790593 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 478405242 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 117707950 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 117707948 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 196929544 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884761683938806e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20570738021888 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚467707645495296 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20917448252928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚401372382052192 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20512191884300 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 196929544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚442391358552064 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚401372399091552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚446790197242368 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20570504339968 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚677 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 281502070 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 478431614 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 281502070 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚421820620530176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 192497864ÒÞºÍ4Ê*/ _wandbruntime‚12221  _runtime‚12221•òcË*Ý 2loss/SRqqt3/assignment_loss‚0.9018662939548935 /loss/FRt3/detection_loss‚1.6323600790346973 trainer/global_step‚10349 2loss/FBt1/assignment_loss‚0.007320721940897913 $loss/total_loss‚42.2089620880099 _step‚851 1loss/SRqqt1/assignment_loss‚1.271994130876966 4loss/FBt2/assignment_loss‚0.00016943310535058993 epoch‚8 _runtime‚12221.580285146 0loss/FRt2/assignment_loss‚3.7050171120222224 0loss/FRt3/assignment_loss‚1.1973892850801349 0loss/SRqqt2/detection_loss‚2.071689491470655 0loss/SRqqt4/detection_loss‚2.266835649426167 0loss/FRt4/assignment_loss‚3.4018168379278744 2loss/SRqqt2/assignment_loss‚1.2065722519117925 1loss/FBt3/assignment_loss‚0.03029589653838344 2loss/FBt4/assignment_loss‚0.059153207768256326 0loss/FRt1/assignment_loss‚2.6988085796093118 /loss/FRt1/detection_loss‚3.0044574830038795 -loss/FRt2/detection_loss‚2.04014732889008 /loss/FRt4/detection_loss‚2.8402004373424194 1loss/SRqqt1/detection_loss‚2.1969801550492263 1loss/SRqqt3/detection_loss‚0.7562839149100946 /loss/FBt1/detection_loss‚2.8854064685957774 /loss/FBt4/detection_loss‚2.5405377691442315 2loss/SRqqt4/assignment_loss‚2.0031576036260677 .loss/FBt2/detection_loss‚2.983233924422945 /loss/FBt3/detection_loss‚0.5072680323575742 % _timestamp‚1.7884761683955443e+09 ÉÉmhÌ*â 0loss/FRt1/assignment_loss‚2.6988085796093118 /loss/FRt1/detection_loss‚3.0044574830038795 -loss/FRt2/detection_loss‚2.04014732889008 /loss/FRt4/detection_loss‚2.8402004373424194 1loss/SRqqt1/detection_loss‚2.1969801550492263 1loss/SRqqt3/detection_loss‚0.7562839149100946 /loss/FBt1/detection_loss‚2.8854064685957774 /loss/FBt4/detection_loss‚2.5405377691442315 2loss/SRqqt4/assignment_loss‚2.0031576036260677 .loss/FBt2/detection_loss‚2.983233924422945 /loss/FBt3/detection_loss‚0.5072680323575742 % _timestamp‚1.7884761683955443e+09 2loss/SRqqt3/assignment_loss‚0.9018662939548935 /loss/FRt3/detection_loss‚1.6323600790346973 trainer/global_step‚10349 2loss/FBt1/assignment_loss‚0.007320721940897913 $loss/total_loss‚42.2089620880099 _step‚851 1loss/SRqqt1/assignment_loss‚1.271994130876966 4loss/FBt2/assignment_loss‚0.00016943310535058993 epoch‚8 _runtime‚12221.580285146 0loss/FRt2/assignment_loss‚3.7050171120222224 0loss/FRt3/assignment_loss‚1.1973892850801349 0loss/SRqqt2/detection_loss‚2.071689491470655 0loss/SRqqt4/detection_loss‚2.266835649426167 0loss/FRt4/assignment_loss‚3.4018168379278744 2loss/SRqqt2/assignment_loss‚1.2065722519117925 1loss/FBt3/assignment_loss‚0.03029589653838344 2loss/FBt4/assignment_loss‚0.059153207768256326ÓvwÍ*‚(:… ‹öçÔÌÆÖ† gpu.0.gpu‚43.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚128.056' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸÀƘšÎ*‚(:õ ‹öçÔë¿’‡ disk.rbd0.out‚ 3.3671875 disk.sda1.out‚0 disk.md0p1.in‚ 10.97265625 disk.rbd2.out‚ 3.3671875 disk.sda.in‚ 5.56640625# disk.nvme0n1.in‚329983.51171875 disk.sda1.in‚0' proc.memory.availableMB‚ 9772.828125 network.recv‚ 4.220357e+06" disk.nvme1n1.in‚201421.6328125" disk.nvme1n1.out‚ 71164.4921875 disk.md0.in‚ 10.97265625# proc.memory.rðoæssMB‚ 3032.23046875! disk.dm-0.out‚144951.12890625 network.sent‚ 2.7789197e+07! disk.sda2.out‚1345.9189453125 disk.rbd0.in‚0.015625 disk.sdb1.in‚0( proc.memory.percent‚18.5072660446167 cpu‚25.19598808136375 disk.dm-0.in‚531405.14453125$ disk./.usageGB‚2973.586887359619 disk.sdb.out‚1345.9189453125 disk.sdb1.out‚0# disk.nvme0n1.out‚73786.63671875$ memory_percent‚40.35139083862305 disk.sda2.in‚ 5.56640625 disk.rbd2.in‚0 disk.sdb.in‚5.40625 proc.cpu.threads‚17 disk.md0p1.out‚ 1297.00390625! disk.sdb2.out‚1345.9189453125 disk.sda.out‚1345.9189453125( disk./.usagePercent‚83.1427339133732 disk.sdb2.in‚5.40625 disk.md0.out‚ 1297.003906256P`¿{Ï*‚(:ð šöçÔÛ™¬‡ disk.sdb.out‚1347.955078125# proc.memory.rssMB‚ 3032.23046875 disk.sdb2.out‚1347.955078125 disk.rbd0.out‚ 3.3671875 disk.sdb1.out‚0 disk.dm-0.out‚ 145022.015625 disk.md0p1.out‚ 1298.98046875 network.sent‚ 2.7899485e+07 disk.md0p1.in‚ 10.97265625 disk.sdb1.in‚0 disk.sda1.in‚0 disk.dm-0.in‚531405.14453125) proc.memory.availableMB‚ 9772.91015625 proc.cpu.threads‚17 disk.sdb2.in‚5.40625) disk./.usagePercent‚83.14466297575717 disk.sda1.out‚0 disk.sda2.in‚ 5.56640625 disk.nvme1n1.out‚ 71235.3125 disk.sdb.in‚5.40625 disk.sda.in‚ 5.56640625% memory_percent‚40.350890159606934 disk.md0.out‚ 1298.98046875# disk.nvme0n1.in‚329983.51171875( proc.memory.percent‚18.5072660446167 cpu‚25.195808798860437 disk.md0.in‚ 10.97265625 network.recv‚ 4.227599e+06" disk.nvme1n1.in‚201421.6328125 disk.sda.out‚1347.955078125 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.rbd2.out‚ 3.3671875! disk.nvme0n1.out‚ 73786.703125% disk./.usageGB‚2973.6558799743652 disk.sda2.out‚1347.955078125aìô‘Ð*‚(:† šöçÔ•’Ô‡ gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚140.754' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.1885 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö·P©‘Ñ*‚(:† ©öçÔÓܼ‡ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚140.478' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.1195 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XŠàÅxÒ*‚(:í ©öçÔ‚—õ‡# disk.nvme1n1.out‚71309.02734375 disk.sdb1.in‚0 disk.sda.out‚1349.3310546875# disk.nvme0n1.out‚73786.88671875 disk.sda2.in‚ 5.5703125 disk.sdb.out‚1349.3310546875( proc.memory.availableMB‚ 9772.8984375$ disk./.usageGB‚2973.780818939209! disk.sda2.out‚1349.3310546875 disk.md0.in‚ 10.9765625 disk.rbd2.in‚0 disk.dm-0.out‚145095.9140625 cpu‚25.195567306880445 disk.rbd0.out‚ 3.3671875 disk.sdb.in‚5.40625 network.recv‚ 4.231663e+06 disk.sdb1.out‚0 disk.dm-0.in‚531405.14453125 disk.sdb2.in‚5.40625# proc.memory.rssMB‚ 3032.23046875 network.sent‚ 2.7905302e+07 disk.sda.in‚ 5.5703125 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0.out‚ 1300.3125 disk.rbd0.in‚0.015625( proc.memory.percent‚18.5072660446167) disk./.usagePercent‚83.14815632150521# disk.nvme0n1.in‚329983.51171875 disk.md0p1.out‚ 1300.3125% memory_percent‚40.350961685180664 proc.cpu.threads‚17 disk.md0p1.in‚ 10.9765625! disk.sdb2.out‚1349.3310546875 disk.rbd2.out‚ 3.3671875" disk.nvme1n1.in‚201421.6328125G ¤F’Ó*‚(:‡ ¸öçÔ“ÛÁ‡ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚138.919' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.72975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëLŒuÔ*‚(:ê ¸öçÔà䢈 disk.md0.out‚ 1301.71484375 disk.dm-0.in‚531405.14453125 disk.dm-0.out‚145172.3359375 disk.sda1.in‚0" memory_percent‚40.350341796875 proc.cpu.threads‚17# disk.nvme1n1.out‚71385.33984375# disk.nvme0n1.in‚329983.51171875! disk.sda2.out‚1350.7841796875 cpu‚25.195368011492068 network.recv‚ 4.235703e+06 disk.rbd2.in‚0 disk.sdb.out‚1350.7841796875 disk.sda.out‚1350.7841796875 disk.sda2.in‚ 5.5703125 network.sent‚ 2.791135e+07" disk.nvme1n1.in‚201421.6328125 disk.md0p1.in‚ 10.9765625 disk.sdb2.in‚5.40625 disk.md0.in‚ 10.9765625 disk.rbd0.in‚0.015625 disk.sda.in‚ 5.5703125! disk.sdb2.out‚1350.7841796875# proc.memory.rssMB‚ 3032.23046875( proc.memory.percent‚18.5072660446167 disk.rbd0.out‚ 3.3671875 disk.sdb.in‚5.40625 disk.sdb1.out‚0 disk.sda1.out‚0) disk./.usagePercent‚83.15450102325848 disk.rbd2.out‚ 3.3671875 disk.sdb1.in‚0# disk.nvme0n1.out‚73786.99609375 disk.md0p1.out‚ 1301.71484375 proc.memory.availableMB‚9773% disk./.usageGB‚2974.0077362060547OX¼ú4Õ*/ _wandbruntime‚12270  _runtime‚12270f úÆÖ*À % lr-AdamW/pg1‚4.81627237562629e-05 % lr-AdamW/pg2‚4.81627237562629e-05 trainer/global_step‚10399 $ _timestamp‚1.788476217192218e+09 _runtime‚12270.377260248 _step‚8522Ë×*Å _step‚852 % lr-AdamW/pg1‚4.81627237562629e-05 % lr-AdamW/pg2‚4.81627237562629e-05 trainer/global_step‚10399 $ _timestamp‚1.788476217192218e+09 _runtime‚12270.377260248ÔÏêÙI4Ø*/ _wandbruntime‚12270  _runtime‚12270K±¥ò'Ù*ìO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 480638323 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚403211369565968 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚12270.378471343 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 118237246 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚444412626032128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚423745636342784 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚423745636342784 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21015518318592 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20608400416906 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 118237248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚423745618339840 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 197825019 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚444412374360576 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21015806840832 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20666756020736 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 197825019 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 193368227 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 282786960 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 197825017 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚423819541001790 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚10399 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚448832711774720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚75131704 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21015806840832 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20666756020736 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 197825019 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 193368227 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 282786960 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 197825017 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚423819541001790 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚10399 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚448832711774720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚75131704 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚12270.378471343 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 118237246 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚444412626032128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚423745636342784 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚423745636342784 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21015518318592 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20608400416906 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 118237248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚423745618339840 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 197825019 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚444412374360576 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 193385964 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 480683316 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚444453624423424 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20668720046592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 480683316 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚403248916962752 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 197843292 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 193386658 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚444453876108288 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20668953728512 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚423859029848200 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚403248899923392 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20668720046592 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20668953728512 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚75138608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚75137916 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _runtime‚12271.329458359 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 282840022 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 197843292 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 197843294 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚423784904376832 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21017797863424 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 480656944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚469892217548288 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21017513548800 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20610129924808 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚423784922379776 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚469891909293568 HBDeviceStatsM~0+|onitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 trainer/global_step‚10399 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚694 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 480656944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚423784922379776 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20610358906100 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 282813652 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 282840022 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚448874395744768 $ _timestamp‚1.788476218143808e+09 _step‚8541^×ð'Ý*úM ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21017513548800 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20610129924808 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚423784922379776 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚469891909293568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 trainer/global_step‚10399 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚694 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 480656944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚423784922379776 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20610358906100 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 282813652 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 282840022 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚448874395744768 $ _timestamp‚1.788476218143808e+09 _step‚854 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 193385964 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 480683316 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚444453624423424 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20668720046592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 480683316 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚403248916962752 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 197843292 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 193386658 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚444453876108288 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20668953728512 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚423859029848200 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚403248899923392 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20668720046592 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20668953728512 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚75138608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚75137916 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _runtime‚12271.329458359 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 282840022 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 197843292 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 197843294 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚423784904376832 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21017797863424 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 480656944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚469892217548288 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457Önzp4Þ*/ _wandbruntime‚12271  _runtime‚12271‡Š?9cß*Ý 2loss/SRqqt1/assignment_loss‚1.2651606375394866 2loss/SRqqt2/assignment_loss‚1.6674801829596981 &loss/total_loss‚41.082999275052515 trainer/global_step‚10399 0loss/FRt3/assignment_loss‚1.3853570729494096 0loss/FRt4/assignment_loss‚2.7586581468582154 1loss/SRqqt4/assignment_loss‚1.843428874015808 1loss/FBt3/assignment_loss‚0.09669244311338518 /loss/FRt1/detection_loss‚2.6019757344867243 /loss/FRt4/detection_loss‚3.6138127624988554 /loss/FRt1/assignment_loss‚2.451485626625292 1loss/SRqqt2/detection_loss‚2.2687908637523653 0loss/FBt3/detection_loss‚0.48543811505660417 0loss/SRqqt4/detection_loss‚2.069992951552073 % _timestamp‚1.7884762181457138e+09 _runtime‚12271.330530644 0loss/FRt2/assignment_loss‚3.3974920672339364 1loss/FBt2/assignment_loss‚0.21834425933715557 /loss/FRt2/detection_loss‚2.2291894714030236 /loss/FRt3/detection_loss‚1.8777292338199914 2loss/SRqqt3/assignment_loss‚1.0349658022296953 1loss/SRqqt1/detection_loss‚1.7708260624907737 1loss/SRqqt3/detection_loss‚0.9051016597998339 .loss/FBt1/detection_loss‚2.798951976829105 /loss/FBt2/detection_loss‚2.0684580522425033 _step‚855 /loss/FBt4/detection_loss‚2.1035623333670874 epoch‚8 1loss/FBt1/assignment_loss‚0.11737219793331456 1loss/FBt4/assignment_loss‚0.05273274695817388E\w¹hà*â 0loss/SRqqt4/detection_loss‚2.069992951552073 % _timestamp‚1.7884762181457138e+09 _runtime‚12271.330530644 0loss/FRt2/assignment_loss‚3.3974920672339364 1loss/FBt2/assignment_loss‚0.21834425933715557 /loss/FRt2/detection_loss‚2.2291894714030236 /loss/FRt3/detection_loss‚1.8777292338199914 2loss/SRqqt3/assignment_loss‚1.0349658022296953 1loss/SRqqt1/detection_loss‚1.7708260624907737 1loss/SRqqt3/detection_loss‚0.9051016597998339 .loss/FBt1/detection_loss‚2.798951976829105 /loss/FBt2/detection_loss‚2.0684580522425033 _step‚855 /loss/FBt4/detection_loss‚2.1035623333670874 epoch‚8 1loss/FBt1/assignment_loss‚0.11737219793331456 1loss/FBt4/assignment_loss‚0.05273274695817388 2loss/SRqqt1/assignment_loss‚1.2651606375394866 2loss/SRqqt2/assignment_loss‚1.6674801829596981 &loss/total_loss‚41.082999275052515 trainer/global_step‚10399 0loss/FRt3/assignment_loss‚1.3853570729494096 0loss/FRt4/assignment_loss‚2.7586581468582154 1loss/SRqqt4/assignment_loss‚1.843428874015808 1loss/FBt3/assignment_loss‚0.09669244311338518 /loss/FRt1/detection_loss‚2.6019757344867243 /loss/FRt4/detection_loss‚3.6138127624988554 /loss/FRt1/assignment_loss‚2.451485626625292 1loss/SRqqt2/detection_loss‚2.2687908637523653 0loss/FBt3/detection_loss‚0.48543811505660417׬Œþá*‚(:… ÇöçÔÝʈ gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚144.212' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.053 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Nr á|â*‚(:ñ ÇöçÔ£ý¯‰" disk.nvme1n1.in‚201421.6328125# disk.nvme0n1.in‚329983.51171875 disk.sda2.in‚ 5.5703125 network.recv‚ 4.239655e+06 disk.sdb1.out‚0" disk.sdb2.out‚1352.28564453125 disk.md0.in‚ 10.9765625% memory_percent‚39.878058433532715 proc.cpu.threads‚17 network.sent‚ 2.791447e+07 disk.sdb.in‚5.40625! disk.sdb.out‚1352.28564453125 disk.md0p1.in‚ 10.9765625 disk.dm-0.out‚ 145459.828125 disk.sda1.out‚0" disk.sda2.out‚1352.28564453125 disk.rbd2.out‚ 3.3671875# proc.memory.rssMB‚ 2952.15234375 disk.nvme1n1.out‚71487.25 disk.sdb1.in‚0! disk.sda.out‚1352.28564453125 disk.sda1.in‚0 cpu‚25.19516597155735) disk./.usagePercent‚83.15671294937363$ disk./.usageGB‚2974.086845397949 disk.sda.in‚ 5.5703125 disk.dm-0.in‚531405.14453125 disk.md0.out‚ 1303.16015625 disk.rbd2.in‚0! disk.nvme0n1.out‚ 73972.578125 disk.rbd0.out‚ 3.3671875 disk.md0p1.out‚ 1303.16015625 disk.sdb2.in‚5.40625 disk.rbd0.in‚0.015625) proc.memory.availableMB‚ 9850.37890625* proc.memory.percent‚18.018507957458496¢ ¡¨‘ã*‚(:† ÖöçÔ­…è… gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚118.142' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.5355 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óÂŒ|ä*‚(:ñ ÖöçÔšª–‡ network.recv‚ 4.246908e+06 disk.md0.out‚ 1305.1015625 disk.rbd0.out‚ 3.3671875" disk.nvme1n1.in‚201421.6328125# proc.memory.rssMB‚ 2953.62109375 proc.cpu.threads‚17) disk./.usagePercent‚83.15671294937363# disk.nvme0n1.in‚329983.51171875 disk.dm-0.out‚ 145564.4375 disk.sda1.out‚0 disk.md0.in‚ 10.9765625 disk.rbd2.out‚ 3.3671875 cpu‚25.19495831817223 disk.md0p1.out‚ 1305.1015625! disk.sdb2.out‚1354.2919921875# disk.nvme1n1.out‚71591.79296875 disk.sdb.out‚1354.2919921875 disk.sdb1.in‚0$ disk./.usageGB‚2974.086845397949 disk.md0p1.in‚ 10.9765625 disk.rbd2.in‚0* proc.memory.percent‚18.027472496032715# disk.nvme0n1.out‚73972.64453125 disk.dm-0.in‚531405.14453125! disk.sda2.out‚1354.2919921875 disk.sdb2.in‚5.40625 disk.sda.in‚ 5.5703125 disk.rbd0.in‚0.015625 disk.sdb1.out‚0 disk.sda.out‚1354.2919921875 disk.sda1.in‚0 disk.sda2.in‚ 5.5703125 disk.sdb.in‚5.40625$ memory_percent‚39.88983631134033 network.sent‚ 2.8022477e+07) proc.memory.availableMB‚ 9848.44921875ÄòxÖ’å*‚(:‡ åöçÔú݆ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚124.661' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.16525 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•ó[V{æ*‚(:ð åöçÔ›¨³‡" disk.nvme1n1.out‚ 71634.8203125! disk.sda2.out‚1356.0791015625 disk.md0.out‚ 1306.8125 disk.sdb.out‚1356.0791015625 disk.sdb1.in‚0 disk.sda.out‚1356.0791015625% disk./.usageGB‚2973.8175010681152 disk.dm-0.in‚531405.14453125 disk.sda1.out‚0 disk.sda2.in‚ 5.5703125 disk.md0p1.out‚ 1306.8125$ memory_percent‚39.88902568817139# proc.memory.rssMB‚ 2953.62109375 disk.md0.in‚ 10.9765625" disk.nvme1n1.in‚201421.6328125 disk.sdb.in‚5.40625# disk.nvme0n1.in‚329983.51171875# disk.nvme0n1.out‚73978.86328125 disk.md0p1.in‚ 10.9765625 network.sent‚ 2.8028294e+07! disk.sdb2.out‚1356.0791015625 disk.rbd2.in‚0 disk.rbd2.out‚ 3.3671875) proc.memory.availableMB‚ 9848.58203125 proc.cpu.threads‚17 disk.sdb2.in‚5.40625 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 3.3671875 disk.sdb1.out‚0! disk.dm-0.out‚145613.68359375 disk.sda1.in‚0) disk./.usagePercent‚83.14918196918211 disk.sda.in‚ 5.5703125* proc.memory.percent‚18.027472496032715 cpu‚25.194761008845017 network.recv‚ 4.251008e+06éKªÚ4ç*/ _wandbruntime‚12325  _runtime‚12325§ŠŽ»Ëè*Å ' lr-AdamW/pg1‚4.5409378348864484e-05 ' lr-AdamW/pg2‚4.5409378348864484e-05 trainer/global_step‚10449 % _timestamp‚1.7884762727080543e+09 _runtime‚12325.893289123 _step‚856Ò”}°Ðé*Ê _step‚856 ' lr-AdamW/pg1‚4.5409378348864484e-05 ' lr-AdamW/pg2‚4.5409378348864484e-05 trainer/global_step‚10449 % _timestamp‚1.7884762727080543e+09 _runtime‚12325.893289123ØiùÏi4ê*/ _wandbruntime‚12325  _runtime‚12325N%Òïò'ë*ìO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20765205395968 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚857 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 198738769 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚425709938192384 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 198738767 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 284151256 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚450916934217216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚405087887437168 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21115406667776 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚720 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 194255787 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚425793996913498 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚425709920189440 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20706338457414 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21115697287168 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚446474891916800 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚425709920189440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚75478439 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 284151256 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚446475143588352 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 194255065 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 % _timestamp‚1.7884762727096217e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚446475143588352 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 284124912 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20706109476330 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 trainer/global_step‚10449 _runtime‚12325.895187292 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 482890025 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 284124912 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚450916910277120 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 198738767 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20764971727360 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 198738769 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0¨È[óì*ñO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚405087904476528 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚472032316944896 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚446474891916800 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch”ÄÆ„%_start/segment.all.allocated‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚446474891916800 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚425709920189440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚75478439 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 284151256 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚446475143588352 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 194255065 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 % _timestamp‚1.7884762727096217e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚446475143588352 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 284124912 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20706109476330 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 trainer/global_step‚10449 _runtime‚12325.895187292 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 482890025 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 284124912 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚450916910277120 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 198738767 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20764971727360 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 198738769 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20765205395968 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚857 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 198738769 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚425709938192384 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 198738767 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 284151256 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚450916934217216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚405087887437168 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21115406667776 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚720 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 194255787 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚425793996913498 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚425709920189440 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20706338457414 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21115697287168 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚425749224229376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20766935753216 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 284177974 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚75485154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚75484498 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚10449 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚425749206226432 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20767169435136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚425833485759908 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚446516141979648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚405125434833952 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 284177974 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 284151604 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21117433447424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 198757042 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 194273304 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 198757044 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 _runtime‚12327.02640996 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚658 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20708067965316 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 482908646 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚446516393664512 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20766935753216 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚450958618187264 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚450958594247168 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 194272646 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚858 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚425749224229376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20708296946608 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 482935018 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 482908646XÁb®'ï*úM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚450958594247168 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 194272646 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚858 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚425749224229376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20708296946608 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 482935018 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 482908646 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 118788148 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 482935018 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚472076027694592 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21117715664896 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚405125417794592 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚425749224229376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20766935753216 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 284177974 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚75485154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚75484498 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚10449 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚425749206226432 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20767169435136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚425833485759908 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚446516141979648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚405125434833952 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 284177974 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 284151604 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21117433447424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 198757042 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 194273304 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 198757044 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 _runtime‚12327.02640996 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚658 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20708067965316 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 482908646 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚446516393664512 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20766935753216 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚450958618187264 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273Úyú¿Þ4ð*/ _wandbruntime‚12327  _runtime‚12327$Z»nañ*Û /loss/FRt4/detection_loss‚3.3364321887493134 1loss/SRqqt1/detection_loss‚1.9690442740917207 /loss/FBt3/detection_loss‚0.6138503632158973 epoch‚8 1loss/SRqqt4/assignment_loss‚2.004578160850898 1loss/FBt4/assignment_loss‚0.24067072818676552 0loss/FBt1/assignment_loss‚0.1644627446266611 2loss/FBt3/assignment_loss‚0.044197166499823366 _runtime‚12327.027964917 /loss/FRt2/detection_loss‚2.0598188085399065 0loss/SRqqt3/detection_loss‚0.792045244735848 1loss/SRqqt4/detection_loss‚2.4178704969261005 /loss/FBt2/detection_loss‚2.7331781658259304 % _timestamp‚1.7884762738428805e+09 0loss/FRt3/assignment_loss‚1.1352349519729614 %loss/total_loss‚41.46962226281726 1loss/SRqqt2/assignment_loss‚1.642704201995262 /loss/FRt3/detection_loss‚1.2097187552601099 /loss/FRt1/assignment_loss‚3.043989600504146 2loss/SRqqt1/assignment_loss‚1.0609586887527258 1loss/SRqqt2/detection_loss‚2.3313406237534116 .loss/FBt1/detection_loss‚2.101151411732038 trainer/global_step‚10449 0loss/FRt2/assignment_loss‚3.2912605615953603 1loss/SRqqt3/assignment_loss‚0.721449602539315 3loss/FBt2/assignment_loss‚0.0021552461813081954 /loss/FBt4/detection_loss‚2.4140980972184076 _step‚859 0loss/FRt4/assignment_loss‚3.3508448779582976 /loss/FRt1/detection_loss‚2.7885673011050507;äfò*à 2loss/SRqqt1/assignment_loss‚1.0609586887527258 1loss/SRqqt2/detection_loss‚2.3313406237534116 .loss/FBt1/detection_loss‚2.101151411732038 trainer/global_step‚10449 0loss/FRt2/assignment_loss‚3.2912605615953603 1loss/SRqqt3/assignment_loss‚0.721449602539315 3loss/FBt2/assignment_loss‚0.0021552461813081954 /loss/FBt4/detection_loss‚2.4140980972184076 _step‚859 0loss/FRt4/assignment_loss‚3.3508448779582976 /loss/FRt1/detection_loss‚2.7885673011050507 /loss/FRt4/detection_loss‚3.3364321887493134 1loss/SRqqt1/detection_loss‚1.9690442740917207 /loss/FBt3/detection_loss‚0.6138503632158973 epoch‚8 1loss/SRqqt4/assignment_loss‚2.004578160850898 1loss/FBt4/assignment_loss‚0.24067072818676552 0loss/FBt1/assignment_loss‚0.1644627446266611 2loss/FBt3/assignment_loss‚0.044197166499823366 _runtime‚12327.027964917 /loss/FRt2/detection_loss‚2.0598188085399065 0loss/SRqqt3/detection_loss‚0.792045244735848 1loss/SRqqt4/detection_loss‚2.4178704969261005 /loss/FBt2/detection_loss‚2.7331781658259304 % _timestamp‚1.7884762738428805e+09 0loss/FRt3/assignment_loss‚1.1352349519729614 %loss/total_loss‚41.46962226281726 1loss/SRqqt2/assignment_loss‚1.642704201995262 /loss/FRt3/detection_loss‚1.2097187552601099 /loss/FRt1/assignment_loss‚3.043989600504146Û¯Ýöšó*‚(:‘ ôöçÔ„ÃÖ† gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚118.474' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.618500000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚#pŠÇ0Ø›Èzô*‚(:ï ôöçÔ“»ü†) disk./.usagePercent‚83.14921023422312% disk./.usageGB‚2973.8185119628906 disk.sda.in‚ 5.5703125 disk.sda.out‚1357.6787109375 disk.dm-0.in‚531405.14453125$ memory_percent‚39.89005088806152 proc.cpu.threads‚17# disk.nvme0n1.in‚329983.51171875" disk.nvme0n1.out‚ 73978.9453125 disk.sda2.in‚ 5.5703125 disk.md0p1.out‚ 1308.3515625 disk.md0.in‚ 10.9765625 disk.dm-0.out‚ 145613.875! disk.sdb2.out‚1357.6787109375 disk.rbd0.in‚0.015625" disk.nvme1n1.out‚ 71634.9296875 disk.sdb.in‚5.40625 disk.sdb.out‚1357.6787109375 disk.md0.out‚ 1308.3515625* proc.memory.percent‚18.027472496032715 network.sent‚ 2.8034245e+07 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 3.3671875 disk.sdb1.out‚0 disk.sda1.in‚0! disk.sda2.out‚1357.6787109375 disk.md0p1.in‚ 10.9765625 disk.sdb2.in‚5.40625( proc.memory.availableMB‚ 9848.4140625 network.recv‚ 4.255208e+06# proc.memory.rssMB‚ 2953.62109375" disk.nvme1n1.in‚201421.6328125 disk.sda1.out‚0 disk.rbd0.out‚ 3.3671875 cpu‚25.194566942000154yó´Žõ*‚(:ƒ ƒ÷çÔÒŒ¢† gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚125.88' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.47 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x$ö*‚(:ö ƒ÷çÔ€¦ô‡# proc.memory.rssMB‚ 2953.62109375 disk.sda1.in‚0 disk.sdb2.in‚5.40625" disk.nvme1n1.in‚201421.6328125% disk./.usageGB‚2974.0921478271484 disk.md0.in‚ 10.9765625 disk.rbd2.in‚0 disk.rbd2.out‚ 3.3671875 disk.dm-0.out‚ 145686.71875* proc.memory.percent‚18.027472496032715) disk./.usagePercent‚83.15686120751326 disk.rbd0.out‚ 3.3671875 disk.sdb.in‚5.40625# disk.nvme0n1.out‚74051.58984375 disk.sda1.out‚0 disk.md0.out‚ 1310.52734375 disk.rbd0.in‚0.015625 disk.sda.in‚ 5.5703125 disk.dm-0.in‚531405.14453125 disk.sda2.in‚ 5.5703125 disk.md0p1.out‚ 1310.52734375 proc.cpu.threads‚17! disk.sda2.out‚1359.9208984375! disk.sdb2.out‚1359.9208984375 disk.sdb.out‚1359.9208984375 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda.out‚1359.9208984375# disk.nvme0n1.in‚329983.51171875 disk.md0p1.in‚ 10.9765625 network.recv‚ 4.26177e+06# disk.nvme1n1.out‚71635.12890625 cpu‚25.194408027276655 network.sent‚ 2.8141501e+07% memory_percent‚39.890265464782715) proc.memory.availableMB‚ 9848.37890625$3Ža’÷*‚(:‡ ’÷çÔŒ§Ö… gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚137.083' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.27075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö•¤þ„ø*‚(:ù ’÷çÔÝú‡ network.sent‚ 2.8147904e+07 disk.dm-0.in‚531405.14453125 disk.sdb2.in‚5.40625 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 3.3671875" disk.nvme1n1.in‚201421.6328125* proc.memory.percent‚18.027472496032715 cpu‚25.194214970463612 disk.sdb1.out‚0 disk.sda1.in‚0 disk.md0p1.out‚ 1312.12890625" disk.sdb2.out‚1361.59130859375! disk.sdb.out‚1361.59130859375% memory_percent‚39.891672134399414# proc.memory.rssMB‚ 2953.62109375 disk.sda.in‚ 5.5703125! disk.sda.out‚1361.59130859375 disk.sda1.out‚0" disk.sda2.out‚1361.59130859375 disk.rbd0.out‚ 3.3671875 disk.rbd2.in‚0 disk.sdb1.in‚0 disk.dm-0.out‚145901.6640625 disk.sda2.in‚ 5.5703125 disk.md0p1.in‚ 10.9765625 disk.sdb.in‚5.40625 proc.cpu.threads‚17) disk./.usagePercent‚83.16391520180514% disk./.usageGB‚2974.3444328308105# disk.nvme0n1.in‚329983.51171875 disk.md0.in‚ 10.9765625! disk.nvme1n1.out‚ 71635.234375( proc.memory.availableMB‚ 9848.1484375 network.recv‚ 4.266645e+06" disk.nvme0n1.out‚ 74266.4296875 disk.md0.out‚ 1312.12890625,BÑ.ù*‚(:… ¡÷çÔ¹ßÞ‡ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚127.446' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.8615 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„ú*‚(:ù ¡÷çÔ´±Çˆ" disk.sdb2.out‚1363.09912109375 disk.md0.in‚ 10.9765625 cpu‚25.19399568306236 disk.sda.in‚ 5.5703125 disk.md0p1.in‚ 10.9765625 disk.sdb1.out‚0) disk./.usagePercent‚83.16738006919111 disk.rbd2.in‚0" disk.nvme1n1.in‚201421.6328125 disk.md0.out‚ 1313.5703125 proc.cpu.threads‚17 network.sent‚ 2.8153723e+07! disk.dm-0.out‚146039.83984375 disk.sdb2.in‚5.40625 network.recv‚ 4.270861e+06 disk.sda1.in‚0 disk.sda2.in‚ 5.5703125! disk.sda.out‚1363.09912109375 disk.md0p1.out‚ 1313.5703125# disk.nvme0n1.out‚74404.26953125" disk.sda2.out‚1363.09912109375 disk.rbd0.in‚0.015625" disk.nvme1n1.out‚ 71635.5703125 disk.sdb.in‚5.40625! disk.sdb.out‚1363.09912109375* proc.memory.percent‚18.027472496032715 disk.rbd0.out‚ 3.3671875 disk.sdb1.in‚0 disk.dm-0.in‚531405.14453125% memory_percent‚39.890241622924805( proc.memory.availableMB‚ 9848.3828125 disk.sda1.out‚0# proc.memory.rssMB‚ 2953.62109375% disk./.usageGB‚2974.4683532714844 disk.rbd2.out‚ 3.3671875# disk.nvme0n1.in‚329983.51171875ø¬§Ä4û*/ _wandbruntime‚12383  _runtime‚12383ÒZEÉü*à & lr-AdamW/pg1‚4.273247704878322e-05 & lr-AdamW/pg2‚4.273247704878322e-05 trainer/global_step‚10499 % _timestamp‚1.7884763306167114e+09 _runtime‚12383.801760681 _step‚860Á ¿cÎý*È & lr-AdamW/pg1‚4.273247704878322e-05 & lr-AdamW/pg2‚4.273247704878322e-05 trainer/global_step‚10499 % _timestamp‚1.7884763306167114e+09 _runtime‚12383.801760681 _step‚860Ü¥€4þ*/ _wandbruntime‚12383  _runtime‚12383$uŠoò'ÿ*ìO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 285489208 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20863187433984 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 199652519 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 119317446 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 285462864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚75827497 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚453001132719616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20804276497922 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚448537661144576 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20863187433984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚75826813 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚448537409473024 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚474216750654976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 195144945 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚448537409473024 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 119317448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 199652519 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 485115381 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20863421102592 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 485141727 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 320850944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21215321024512 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚406964422347728 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20804047516838 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚448537661144576 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚474216429804032 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚12383.803150648 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚686 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚427768452825206 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚427674240041984 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚453001108779520 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21215617935360 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚427768698845650 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 65619886083_Û•÷'€+ñO QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚406964422347728 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20804047516838 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚448537661144576 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚474216429804032 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚12383.803150648 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚686 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚427768452825206 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚427674240041984 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚453001108779520 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21215617935360 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚427768698845650 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 285489208 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20863187433984 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 199652519 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 119317446 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 285462864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚75827497 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚453001132719616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20804276497922 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚448537661144576 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20863187433984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚75826813 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚448537409473024 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚474216750654976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 195144945 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚448537409473024 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 119317448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 199652519 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 485115381 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20863421102592 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 485141727 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 320850944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21215321024512 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712Ý©Á—v4+/ _wandbruntime‚12384  _runtime‚12384z`ÃL¿‚+õM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚448578659535872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 285515926 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20865385141760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚75834196 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚666 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 199670794 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚448578911220736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚427713508076032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 485186720 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 199670792 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 485160348 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20865151459840 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 285489556 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 195161778 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20806234987116 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚427713526078976 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 K=DeviceStatsMonitor.on_train_batch_end/active.larõï h<ge_pool.freed‚ 199670792 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 285489556 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚427713508076032 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 119328250 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 485186720 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚10499 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚427808187692268 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 199670794 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚427713526078976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚474260455419392 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚427807941671616 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚862 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21217638729728 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 195162446 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚474260145067520 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 485186720 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 199670792 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 485160348 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20865151459840 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 285489556 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 195161778 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20806234987116 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚427713526078976 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 199670792 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 285489556 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚427713508076032 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 119328250 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 485186720 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚10499 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚427808187692268 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 199670794 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚427713526078976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚474260455419392 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚427807941671616 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚862 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21217638729728 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 195162446 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚474260145067520 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136Þôð°"4„+/ _wandbruntime‚12384  _runtime‚12384 ƒÿY…+Ó .loss/FRt4/detection_loss‚2.892370773686303 1loss/SRqqt3/detection_loss‚0.8526503713995529 _runtime‚12384.810106304 0loss/SRqqt1/detection_loss‚2.099715448561169 %loss/total_loss‚41.32382109241334 epoch‚8 _step‚863 0loss/FRt2/assignment_loss‚2.8031808137893677 0loss/SRqqt2/detection_loss‚2.733421856448764 1loss/SRqqt4/detection_loss‚2.2496963876944323 .loss/FBt1/detection_loss‚2.261364409152199 trainer/global_step‚10499 /loss/FRt1/assignment_loss‚2.663640565731946 2loss/SRqqt1/assignment_loss‚1.6484753061972914 /loss/FRt3/detection_loss‚1.3760331433029875 % _timestamp‚1.7884763316251607e+09 0loss/FBt2/assignment_loss‚0.2537115600923729 0loss/FBt3/assignment_loss‚0.1351788738279193 1loss/FBt4/assignment_loss‚0.32001123366145634 .loss/FBt2/detection_loss‚2.346541987998145 /loss/FBt4/detection_loss‚2.1630503279822215 2loss/SRqqt2/assignment_loss‚1.6192793855887084 .loss/FRt1/detection_loss‚2.608089036801282 .loss/FBt3/detection_loss‚0.397104072721528 2loss/SRqqt4/assignment_loss‚1.5615581126453786 0loss/FRt3/assignment_loss‚1.5018185490653628 0loss/FRt4/assignment_loss‚3.4275142351786294 1loss/SRqqt3/assignment_loss‚1.092811078376447 0loss/FBt1/assignment_loss‚0.1061515423138497 /loss/FRt2/detection_loss‚2.2104520201960223ïŸWA^†+Ø 0loss/FBt3/assignment_loss‚0.1351788738279193 1loss/FBt4/assignment_loss‚0.32001123366145634 .loss/FBt2/detection_loss‚2.346541987998145 /loss/FBt4/detection_loss‚2.1630503279822215 2loss/SRqqt2/assignment_loss‚1.6192793855887084 .loss/FRt1/detection_loss‚2.608089036801282 .loss/FBt3/detection_loss‚0.397104072721528 2loss/SRqqt4/assignment_loss‚1.5615581126453786 0loss/FRt3/assignment_loss‚1.5018185490653628 0loss/FRt4/assignment_loss‚3.4275142351786294 1loss/SRqqt3/assignment_loss‚1.092811078376447 0loss/FBt1/assignment_loss‚0.1061515423138497 /loss/FRt2/detection_loss‚2.2104520201960223 .loss/FRt4/detection_loss‚2.892370773686303 1loss/SRqqt3/detection_loss‚0.8526503713995529 _runtime‚12384.810106304 0loss/SRqqt1/detection_loss‚2.099715448561169 %loss/total_loss‚41.32382109241334 epoch‚8 _step‚863 0loss/FRt2/assignment_loss‚2.8031808137893677 0loss/SRqqt2/detection_loss‚2.733421856448764 1loss/SRqqt4/detection_loss‚2.2496963876944323 .loss/FBt1/detection_loss‚2.261364409152199 trainer/global_step‚10499 /loss/FRt1/assignment_loss‚2.663640565731946 2loss/SRqqt1/assignment_loss‚1.6484753061972914 /loss/FRt3/detection_loss‚1.3760331433029875 % _timestamp‚1.7884763316251607e+09 0loss/FBt2/assignment_loss‚0.2537115600923729ßä–UI‘‡+‚(:† °÷çÔçáɆ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚118.142' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.5355 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜rÙˆ+‚(:ö °÷çÔ½‘è†" disk.nvme0n1.out‚ 74552.5546875 disk.sdb2.in‚5.40625 disk.rbd2.in‚0# disk.nvme1n1.out‚71635.72265625 disk.dm-0.in‚531405.14453125 disk.sda1.out‚0$ proc.memory.availableMB‚9848.125! disk.sdb.out‚1364.49365234375 disk.md0.in‚ 10.9765625 disk.sdb1.in‚0! disk.dm-0.out‚146188.27734375 disk.sda1.in‚0 disk.rbd2.out‚ 3.3671875 disk.sdb1.out‚0! disk.sda.out‚1364.49365234375# disk.nvme0n1.in‚329983.51171875" disk.sdb2.out‚1364.49365234375* proc.memory.percent‚18.027472496032715 disk.sdb.in‚5.40625" disk.sda2.out‚1364.49365234375# proc.memory.rssMB‚ 2953.62109375 cpu‚25.193809056098512 disk.rbd0.out‚ 3.3671875 disk.sda2.in‚ 5.5703125% memory_percent‚39.891815185546875 proc.cpu.threads‚17% disk./.usageGB‚2974.5934104919434 disk.md0.out‚ 1314.8984375 disk.sda.in‚ 5.5703125 disk.md0p1.in‚ 10.9765625 disk.md0p1.out‚ 1314.8984375 network.sent‚ 2.8159791e+07 network.recv‚ 4.275257e+06) disk./.usagePercent‚83.17087672141567 disk.rbd0.in‚0.015625" disk.nvme1n1.in‚201421.6328125ìÐ3œ‰+‚(:‘ ¿÷çÔôÁ†† gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·>x…Š+‚(:ú ¿÷çÔº”„ˆ disk.sdb.in‚5.40625$ memory_percent‚39.88964557647705" disk.nvme1n1.in‚201421.6328125! disk.sdb.out‚1366.36962890625# proc.memory.rssMB‚ 2953.62109375 proc.cpu.threads‚17 disk.md0p1.out‚ 1316.70703125 network.sent‚ 2.8268825e+07 disk.sda1.in‚0 disk.md0.in‚ 10.9765625 disk.sdb1.out‚0 cpu‚25.193610434557144 disk.sda1.out‚0 disk.sda2.in‚ 5.5703125 disk.rbd0.in‚0.015625 network.recv‚ 4.281619e+06# disk.nvme0n1.in‚329983.51171875 disk.sdb1.in‚0* proc.memory.percent‚18.027472496032715$ disk./.usageGB‚2975.095474243164! disk.dm-0.out‚146445.14453125 disk.rbd0.out‚ 3.3671875! disk.sda.out‚1366.36962890625( disk./.usagePercent‚83.1849146340298 disk.sdb2.in‚5.40625" disk.sdb2.out‚1366.36962890625 disk.md0.out‚ 1316.70703125 disk.rbd2.in‚0 disk.sda.in‚ 5.5703125) proc.memory.availableMB‚ 9848.48046875# disk.nvme0n1.out‚74809.21484375 disk.dm-0.in‚531405.14453125" disk.sda2.out‚1366.36962890625 disk.md0p1.in‚ 10.9765625 disk.rbd2.out‚ 3.3671875" disk.nvme1n1.out‚ 71635.9296875dÈúÛ‘‹+‚(:† Î÷çÔοý‡ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚105.111' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚26.27775 gpu.0.smClock‚1200 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Qqò%wŒ+‚(:ì Î÷çÔ§‡“ˆ# disk.nvme0n1.in‚329983.51171875 disk.rbd0.out‚ 3.3671875* proc.memory.percent‚18.027472496032715 network.sent‚ 2.8273067e+07 disk.md0p1.in‚ 10.9765625 disk.rbd2.out‚ 3.3671875 disk.sdb.out‚1367.892578125 disk.sda.in‚ 5.5703125 disk.md0.out‚ 1318.171875" disk.nvme1n1.out‚ 71752.4453125 disk.sda.out‚1367.892578125" disk.nvme0n1.out‚ 74982.9140625 disk.dm-0.in‚531405.14453125 disk.md0p1.out‚ 1318.171875 disk.rbd2.in‚0" disk.nvme1n1.in‚201421.6328125 disk.sdb1.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 9848.3671875 network.recv‚ 4.286205e+06) disk./.usagePercent‚83.18835560945658 disk.sda2.in‚ 5.5703125 disk.sdb2.in‚5.40625 disk.md0.in‚ 10.9765625 disk.sdb1.out‚0 disk.sda1.in‚0% disk./.usageGB‚2975.2185401916504 disk.sda2.out‚1367.892578125 disk.rbd0.in‚0.015625# proc.memory.rssMB‚ 2953.62109375 disk.sdb2.out‚1367.892578125 disk.dm-0.out‚ 146735.359375% memory_percent‚39.890336990356445 cpu‚25.19339681225629 proc.cpu.threads‚17 disk.sdb.in‚5.40625"9 éw+‚(:ì Ý÷çÔëïˇ network.sent‚ 2.8280988e+07! disk.nvme0n1.out‚ 75228.765625( disk./.usagePercent‚83.1848582106083 disk.sda.out‚1369.6005859375 disk.sda2.in‚ 5.5703125 disk.md0.in‚ 10.9765625 disk.rbd0.out‚ 3.3671875# disk.nvme1n1.out‚71752.91796875 disk.sda.in‚ 5.5703125! disk.sda2.out‚1369.6005859375 disk.sdb2.in‚5.40625 disk.md0.out‚ 1319.8125 disk.rbd0.in‚0.015625% memory_percent‚39.891910552978516% disk./.usageGB‚2975.0934562683105 disk.sda1.in‚0 disk.rbd2.in‚0 disk.sdb.in‚5.40625 proc.cpu.threads‚17# disk.nvme0n1.in‚329983.51171875 disk.dm-0.in‚531405.14453125 disk.sdb.out‚1369.6005859375 disk.sdb1.out‚0 disk.sda1.out‚0" disk.nvme1n1.in‚201421.6328125 disk.sdb1.in‚0# proc.memory.rssMB‚ 2953.62109375* proc.memory.percent‚18.027472496032715 cpu‚25.193226614027648 network.recv‚ 4.28998e+06 disk.md0p1.in‚ 10.9765625! disk.sdb2.out‚1369.6005859375 disk.rbd2.out‚ 3.3671875' proc.memory.availableMB‚ 9848.109375! disk.dm-0.out‚146981.68359375 disk.md0p1.out‚ 1319.8125Z‚z’Ž+‚(:‡ Ý÷çÔ÷Ʋœ gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚141.087' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.27175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 µ?e4+/ _wandbruntime‚12442  _runtime‚12442/vÉÉ+à trainer/global_step‚10549 % _timestamp‚1.7884763892154186e+09 _runtime‚12442.400401591 _step‚864 & lr-AdamW/pg1‚4.013257771823787e-05 & lr-AdamW/pg2‚4.013257771823787e-05›Ð Α+È _step‚864 & lr-AdamW/pg1‚4.013257771823787e-05 & lr-AdamW/pg2‚4.013257771823787e-05 trainer/global_step‚10549 % _timestamp‚1.7884763892154186e+09 _runtime‚12442.400401591àr®Ã4’+/ _wandbruntime‚12442  _runtime‚12442sŒUC#“+ìO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚450600178700800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 200566269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚408840940218928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚476400881285120 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20902214538430 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 % _timestamp‚1.7884763892166078e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 119857548 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚429742908736914 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20961403140608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _step‚865 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚76172204 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚450599927029248 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚455085331222016 _runtime‚12442.401876668 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20961636809216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 487393429 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚429638523888640 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 200566267 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20961636809216 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21315550063104 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 200566267 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20961403140608 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚429638523888640 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 196030448 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚429743154757358 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 119857546 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21315261540864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚450600178700800 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚696 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚476400568822784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚450599927029248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 286827160 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 286800816 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 200566269 N@Der9t…ÚviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 196029750 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20901985557346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚455085307281920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 286800816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚429638541891584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚76172900 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 trainer/global_step‚10549 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006Œ«÷'”+ñO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚450600178700800 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚696 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚476400568822784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚450599927029248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 286827160 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 286800816 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 200566269 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 196029750 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20901985557346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚455085307281920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 286800816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚429638541891584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚76172900 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 trainer/global_step‚10549 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚450600178700800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 200566269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚408840940218928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚476400881285120 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20902214538430 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 % _timestamp‚1.7884763892166078e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 119857548 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚429742908736914 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20961403140608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _step‚865 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚76172204 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚450599927029248 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚455085331222016 _runtime‚12442.401876668 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20961636809216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 487393429 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚429638523888640 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 200566267 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20961636809216 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21315550063104 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 200566267 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20961403140608 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚429638523888640 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 196030448 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚429743154757358 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 119857546 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21315261540864á@eX04•+/ _wandbruntime‚12443  _runtime‚12443{¢,šü&–+öM FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 487438422 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚76180433 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚476444560277504 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 487412050 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 % _timestamp‚1.7884763904220304e+09 _runtime‚12443.609035983 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20963600848384 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 487412050 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚76179762 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20963367166464 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 286853878 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 200584542 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20963367166464 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 200584542 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 196048110 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚455126991251968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20903944046332 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104ÝIèb'—+ûM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 200584544 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚476444243634176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 200584544 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 286827508 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20963367166464 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 286853878 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 200584542 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20963367166464 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 200584542 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 196048110 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚455126991251968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20903944046332 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 487438422 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚76180433 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚476444560277504 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 487412050 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 % _timestamp‚1.7884763904220304e+09 _runtime‚12443.609035983 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20963600848384 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 487412050 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚76179762 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚452662444585472 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 288165112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚76521549 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚457169505784320 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 196918540 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 489645131 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299879424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚452662696257024 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21059852515840 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 201480019 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21000152578938 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 288165112 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21059618847232 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚431602825738240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚12503.163760572 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 120397648 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚410717458090128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚431602843741184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21415157594624 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚431717364648622 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚478584963258368 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚657 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21059618847232 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚457169529724416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608×a]÷'¨+ñO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299879424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚452662696257024 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21059852515840 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 201480019 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21000152578938 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 288165112 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21059618847232 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚431602825738240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚12503.163760572 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 120397648 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚410717458090128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚431602843741184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21415157594624 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚431717364648622 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚478584963258368 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚657 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21059618847232 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚457169529724416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚452662696257024 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 196919197 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚431717610669066 trainer/global_step‚10599 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 201480017 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚431602843741184 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚478584663378944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚410717441050768 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚452662444585472 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 288165112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚76521549 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚457169505784320 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 196918540 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 489645131åBãà"4©+/ _wandbruntime‚12504  _runtime‚12504e±Ö°ñª+öM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 _step‚870 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚431642111775232 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 201498292 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 196936553 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚410754988447552 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884764511834223e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 489690124 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚452703946333184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 489690124 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚76528105 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21417167654400 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚452703694648320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 196937243 trainer/global_step‚10599 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚410754971408192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _runtime‚12504.370679861 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚688 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚452703946333184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚431756853495032 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 201498294 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21061582873088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚76528793 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21001882086840 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 489663752 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 120408450 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21417456163328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 201498292 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚478628357408768 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21002111068132 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚457211189754368 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 288191830 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 489663752 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚431642129778176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚431642129778176 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚431642111775232 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚76528793 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21001882086840 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 489663752 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 120408450 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21417456163328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 201498292 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚478628357408768 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21002111068132 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚457211189754368 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 288191830 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 489663752 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚431642129778176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚431642129778176 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚431642111775232 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 _step‚870 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚431642111775232 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 201498292 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 196936553 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚410754988447552 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884764511834223e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 489690124 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚452703946333184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 489690124 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚76528105 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21417167654400 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚452703694648320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 196937243 trainer/global_step‚10599 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚410754971408192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _runtime‚12504.370679861 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚688 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚452703946333184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚431756853495032 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 201498294 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21061582873088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21157834553856 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚433691820560330 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 202393767 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 197806981 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚454725213813248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚76869233 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 120937748 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚412593958921968 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 491896833 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚433692066580774 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21158068222464 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚433567145590784 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21515390678528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚459253728226816 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚459253704286720 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21097861638362 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 202393767 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚454724962141696 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 289476720 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104´ p÷'¼+ñO _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21098090619446 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21157834553856 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚433691820560330 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 202393767 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 197806981 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚454725213813248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚76869233 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 120937748 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚412593958921968 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 491896833 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚433692066580774 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21158068222464 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚433567145590784 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21515390678528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚459253728226816 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚459253704286720 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21097861638362 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 202393767 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚454724962141696 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 289476720 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 202393769 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚433567145590784 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚480768810637312 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21158068222464 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 trainer/global_step‚10649 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚454724962141696 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21157834553856 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 491870487 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 289503064 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚454725213813248 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 197806277 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚433567127587840 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 289503064 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 120937746 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚76868531 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚412593975961328 _step‚873 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 % _timestamp‚1.7884765107245486e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚704éyn{e4½+/ _wandbruntime‚12565  _runtime‚12565úKwgû&¾+õM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚687 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 202412042 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 120948550 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 491915454 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚433606431627776 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚76875802 trainer/global_step‚10649 _runtime‚12565.13243776 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21517385930240 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21100049108640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚454766463889408 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚874 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21517099518464 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 289503412 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚685 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚480812487775232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚433606431627776 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21160032261632 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚433606413624832 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 202412044 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚480812798127104 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 202412044 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 491941826 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚454766463889408 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 197824352 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 491941826 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 289529782 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 289503412 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚454766212204544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 289529782 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚459295388256768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21159798579712 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 120948548 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 % _timestamp‚1.7884765119452524e+09 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 197823665 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚412631506318752 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚433606413624832 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21099820127348 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104^…ëù#¿+úM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚433606413624832 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 202412044 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚480812798127104 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 202412044 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 491941826 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚454766463889408 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 197824352 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 491941826 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 289529782 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 289503412 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚454766212204544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 289529782 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚459295388256768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21159798579712 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 120948548 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 % _timestamp‚1.7884765119452524e+09 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 197823665 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚412631506318752 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚433606413624832 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21099820127348 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚687 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 202412042 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 120948550 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 491915454 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚433606431627776 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚76875802 trainer/global_step‚10649 _runtime‚12565.13243776 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21517385930240 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21100049108640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚454766463889408 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚874 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1091é)71712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21517099518464 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 289503412 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚685 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚480812487775232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚433606431627776 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21160032261632êÙÒ¿4À+/ _wandbruntime‚12565  _runtime‚125659™ŠBYÁ+Ó _step‚875 2loss/SRqqt4/assignment_loss‚1.2412733775563538 -loss/FBt1/detection_loss‚2.20196940501531 %loss/total_loss‚42.90563631162477 /loss/FBt3/detection_loss‚0.5442295605083928 0loss/FRt3/assignment_loss‚1.2829430452982584 2loss/SRqqt2/assignment_loss‚1.5961244435360034 1loss/FBt1/assignment_loss‚0.11298398622590373 % _timestamp‚1.7884765119490328e+09 2loss/SRqqt3/assignment_loss‚0.9687306784614279 1loss/FBt2/assignment_loss‚0.09987313298430915 /loss/FBt4/detection_loss‚1.8959751803624003 /loss/FRt1/assignment_loss‚2.536339592933655 0loss/SRqqt4/detection_loss‚2.580897762129704 _runtime‚12565.133800026 /loss/FRt4/assignment_loss‚3.255963307619095 1loss/FBt4/assignment_loss‚0.05551428880880063 .loss/FRt1/detection_loss‚2.734060710668564 .loss/FBt2/detection_loss‚3.763414596517881 trainer/global_step‚10649 2loss/SRqqt1/assignment_loss‚1.5101460592448712 .loss/FRt4/detection_loss‚3.358660897612572 0loss/SRqqt1/detection_loss‚2.707352591753006 0loss/SRqqt2/detection_loss‚2.811174894372622 epoch‚8 /loss/FRt2/assignment_loss‚3.172991971556957 1loss/FBt3/assignment_loss‚0.08106380105112253 .loss/FRt2/detection_loss‚1.985511176294844 /loss/FRt3/detection_loss‚1.5206699485828479 1loss/SRqqt3/detection_loss‚0.8877719025298649Vü‚^Â+Ø /loss/FRt2/assignment_loss‚3.172991971556957 1loss/FBt3/assignment_loss‚0.08106380105112253 .loss/FRt2/detection_loss‚1.985511176294844 /loss/FRt3/detection_loss‚1.5206699485828479 1loss/SRqqt3/detection_loss‚0.8877719025298649 _step‚875 2loss/SRqqt4/assignment_loss‚1.2412733775563538 -loss/FBt1/detection_loss‚2.20196940501531 %loss/total_loss‚42.90563631162477 /loss/FBt3/detection_loss‚0.5442295605083928 0loss/FRt3/assignment_loss‚1.2829430452982584 2loss/SRqqt2/assignment_loss‚1.5961244435360034 1loss/FBt1/assignment_loss‚0.11298398622590373 % _timestamp‚1.7884765119490328e+09 2loss/SRqqt3/assignment_loss‚0.9687306784614279 1loss/FBt2/assignment_loss‚0.09987313298430915 /loss/FBt4/detection_loss‚1.8959751803624003 /loss/FRt1/assignment_loss‚2.536339592933655 0loss/SRqqt4/detection_loss‚2.580897762129704 _runtime‚12565.133800026 /loss/FRt4/assignment_loss‚3.255963307619095 1loss/FBt4/assignment_loss‚0.05551428880880063 .loss/FRt1/detection_loss‚2.734060710668564 .loss/FBt2/detection_loss‚3.763414596517881 trainer/global_step‚10649 2loss/SRqqt1/assignment_loss‚1.5101460592448712 .loss/FRt4/detection_loss‚3.358660897612572 0loss/SRqqt1/detection_loss‚2.707352591753006 0loss/SRqqt2/detection_loss‚2.811174894372622 epoch‚8ëýÐ/Ù’Ã+‚(:‡ äøçÔ̇ˆ gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07!Ò—Ä+‚(:ö äøçÔ¼œÝˆ disk.md0.out‚ 1335.94140625# disk.nvme0n1.out‚76810.55859375 disk.sda.out‚1386.2998046875) disk./.usagePercent‚83.18485597073713 disk.rbd0.out‚ 3.3671875 disk.rbd2.out‚ 3.3671875 disk.sdb.in‚5.40625 cpu‚25.191429791801863 disk.md0p1.out‚ 1335.94140625 disk.dm-0.in‚531413.62109375 proc.cpu.threads‚17 disk.sdb2.in‚5.40625( proc.memory.availableMB‚ 9848.2421875$ disk./.usageGB‚2975.093376159668 disk.sda1.in‚0 disk.sdb1.in‚0# disk.nvme0n1.in‚329984.59765625# proc.memory.rssMB‚ 2953.62109375* proc.memory.percent‚18.027472496032715 disk.sdb.out‚1386.2998046875 disk.sda.in‚ 5.57421875 network.recv‚ 4.33257e+06 disk.sda2.in‚ 5.57421875! disk.sdb2.out‚1386.2998046875" disk.nvme1n1.in‚201429.0234375! disk.dm-0.out‚148564.99609375 network.sent‚ 2.8547296e+07 disk.sda1.out‚0 disk.md0p1.in‚ 10.98046875 disk.rbd2.in‚0! disk.sda2.out‚1386.2998046875 disk.md0.in‚ 10.98046875 disk.rbd0.in‚0.015625 disk.nvme1n1.out‚ 71754.4375 disk.sdb1.out‚0$ memory_percent‚39.89109992980957ÔT_.›Å+‚(: óøçÔˆï؇ gpu.0.gpu‚8.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚106.329' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚26.582249999999995 gpu.0.smClock‚1200 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’°ß¦VÆ+‚(:Ë óøçÔªÀŽˆ disk.sda2.out‚1388$ memory_percent‚39.88981246948242 proc.cpu.threads‚17) disk./.usagePercent‚83.17716883952659 disk.sda2.in‚ 5.57421875 disk.sdb2.in‚5.40625 disk.rbd0.in‚0.015625 disk.rbd2.in‚0" disk.nvme1n1.in‚201429.0234375 disk.sdb1.out‚0* proc.memory.percent‚18.027472496032715 network.sent‚ 2.8645226e+07 network.recv‚ 4.339643e+06 disk.sda1.in‚0 disk.md0p1.out‚ 1337.56640625 disk.dm-0.in‚531413.62109375 disk.md0.out‚ 1337.56640625 disk.rbd2.out‚ 3.3671875! disk.nvme1n1.out‚ 71754.546875 cpu‚25.19124443224221$ disk./.usageGB‚2974.818447113037# disk.nvme0n1.out‚76973.53515625 disk.md0.in‚ 10.98046875 disk.rbd0.out‚ 3.3671875# proc.memory.rssMB‚ 2953.62109375 disk.sda.out‚1388! disk.dm-0.out‚148728.08203125 disk.md0p1.in‚ 10.98046875 disk.sdb2.out‚1388 disk.sdb.out‚1388 disk.sda.in‚ 5.57421875# disk.nvme0n1.in‚329984.59765625 disk.sda1.out‚0 disk.sdb.in‚5.40625 disk.sdb1.in‚0' proc.memory.availableMB‚ 9848.453125iSú8’Ç+‚(:‡ ‚ùçÔõꌆ gpu.0.gpu‚14.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚128.665' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.16625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õôÊòvÈ+‚(:ë ‚ùçÔ’Ô㇠network.recv‚ 4.34432e+06 disk.md0p1.in‚ 10.98046875 disk.md0.in‚ 10.98046875# disk.nvme0n1.in‚329984.59765625$ memory_percent‚39.89255428314209* proc.memory.percent‚18.027472496032715 cpu‚25.19103854868382 disk.md0p1.out‚ 1339.65234375 disk.rbd0.out‚ 3.3671875 disk.nvme1n1.out‚ 71754.6875 disk.sda.in‚ 5.57421875! disk.dm-0.out‚148824.70703125) disk./.usagePercent‚83.17687829623709% disk./.usageGB‚2974.8080558776855 disk.sdb.in‚5.40625 disk.sdb1.out‚0 disk.sda1.out‚0 disk.sda2.in‚ 5.57421875 disk.rbd0.in‚0.015625" disk.nvme1n1.in‚201429.0234375 disk.sdb.out‚ 1390.1640625 proc.cpu.threads‚17 disk.sda1.in‚0# disk.nvme0n1.out‚77070.01953125 disk.rbd2.out‚ 3.3671875 disk.sda.out‚ 1390.1640625 disk.dm-0.in‚531413.62109375 network.sent‚ 2.8649301e+07 disk.sdb2.in‚5.40625 disk.sdb1.in‚0# proc.memory.rssMB‚ 2953.62109375 disk.sda2.out‚ 1390.1640625 disk.sdb2.out‚ 1390.1640625 disk.md0.out‚ 1339.65234375 disk.rbd2.in‚0) proc.memory.availableMB‚ 9848.003906254ê€k‘É+‚(:† ‘ùçÔÓÜä‡ gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚102.658' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.6645 gpu.0.smClock‚1170 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ‹à1Ê+‚(:ô ‘ùçÔÜÖLj disk.sdb.in‚5.40625 disk.sdb1.in‚0# disk.nvme0n1.in‚329984.59765625" disk.sda2.out‚1391.81884765625" disk.sdb2.out‚1391.81884765625* proc.memory.percent‚18.027472496032715 network.sent‚ 2.8656394e+07 disk.rbd2.out‚ 3.3671875 disk.sdb1.out‚0 disk.dm-0.out‚148967.6796875 disk.sda.in‚ 5.57421875 disk.md0.out‚1341.25 disk.rbd0.out‚ 3.3671875% memory_percent‚39.890265464782715) proc.memory.availableMB‚ 9848.37890625# proc.memory.rssMB‚ 2953.62109375 cpu‚25.19084828657855 disk.rbd2.in‚0! disk.sda.out‚1391.81884765625 disk.dm-0.in‚531413.62109375) disk./.usagePercent‚83.17786767933305 disk.rbd0.in‚0.015625 disk.sda1.out‚0 disk.md0p1.in‚ 10.98046875 disk.sdb2.in‚5.40625 proc.cpu.threads‚17! disk.sdb.out‚1391.81884765625# disk.nvme0n1.out‚77142.13671875 disk.md0.in‚ 10.98046875 network.recv‚ 4.347926e+06% disk./.usageGB‚2974.8434410095215# disk.nvme1n1.out‚71825.54296875 disk.sda1.in‚0 disk.sda2.in‚ 5.57421875 disk.md0p1.out‚1341.25" disk.nvme1n1.in‚201429.0234375Q™Ì‘4Ë+/ _wandbruntime‚12625  _runtime‚12625ÊoYÊÌ+Ä _runtime‚12625.137402066 _step‚876 ' lr-AdamW/pg1‚3.2800228785465004e-05 ' lr-AdamW/pg2‚3.2800228785465004e-05 trainer/global_step‚10699 $ _timestamp‚1.788476571952268e+09}£µBÏÍ+É $ _timestamp‚1.788476571952268e+09 _runtime‚12625.137402066 _step‚876 ' lr-AdamW/pg1‚3.2800228785465004e-05 ' lr-AdamW/pg2‚3.2800228785465004e-05 trainer/global_step‚10699ì ¨ëÅ4Î+/ _wandbruntime‚12625  _runtime‚12625Ó!BZò'Ï+ìO [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21195799678870 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚77218837 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21615226774016 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚435531429437440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚482952849429504 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 203307517 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 290814672 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 494148535 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 121477848 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 494122189 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 203307517 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21256050260480 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚461337926729216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚435666276472038 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21256283929088 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 203307519 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21256050260480 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280133632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 290841016 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚456787479697920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21196028659954 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 trainer/global_step‚10699 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚673 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚435531447440384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 290814672 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21256283929088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0¦k„E÷'Ð+ñO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 494122189 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 203307517 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21256050260480 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚461337926729216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚435666276472038 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21256283929088 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 203307519 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21256050260480 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280133632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 290841016 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚456787479697920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21196028659954 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 trainer/global_step‚10699 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚673 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚435531447440384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 290814672 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21256283929088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21195799678870 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚77218837 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21615226774016 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚435531429437440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚482952849429504 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 203307517 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 290814672 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 494148535 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 121477848 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300íI)aÎ4Ñ+/ _wandbruntime‚12626  _runtime‚12626é°jHÒ+õM E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚12626.36067999 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 290867734 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚456828981445632 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 198714490 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚435705765318448 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21258247968256 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚435570733477376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚461379586759168 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21197758167856 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 494193528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 494193528 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21258014286336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚482996837799936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚686 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚435570715474432 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21616936494592 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21617227100672 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 203325792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 494167156 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 290841364 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚435570733477376 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 290841364 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _step‚878 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21258014286336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 % _timestamp‚1.7884765731735377e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 494167156 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 198713804 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚435706011339100 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 203325792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 trainer/global_step‚10699 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚456828981445632 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚482996523253760 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚461379610699264 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0d¶C'Ó+úM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21258014286336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚482996837799936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚686 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚435570715474432 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21616936494592 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21617227100672 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 203325792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 494167156 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 290841364 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚435570733477376 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 290841364 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _step‚878 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21258014286336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 % _timestamp‚1.7884765731735377e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 494167156 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 198713804 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚435706011339100 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 203325792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 trainer/global_step‚10699 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚456828981445632 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚482996523253760 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚461379610699264 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚12626.36067999 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 290867734 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚456828981445632 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 198714490 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚435705765318448 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21258247968256 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚435570733477376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚461379586759168 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21197758167856 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 494193528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 494193528îFš4Ô+/ _wandbruntime‚12626  _runtime‚12626_ûù]Õ+× 2loss/FBt1/assignment_loss‚0.023536896591085425 .loss/FRt3/detection_loss‚1.387158755833904 /loss/FBt1/detection_loss‚2.5076894793245526 %loss/total_loss‚43.01349350384428 2loss/SRqqt4/assignment_loss‚1.2784093455834822 .loss/FRt4/detection_loss‚2.718955032135311 0loss/SRqqt1/detection_loss‚1.778451959292094 % _timestamp‚1.7884765731772368e+09 0loss/FRt1/assignment_loss‚3.0012282440739293 1loss/SRqqt1/assignment_loss‚1.238123903943536 2loss/SRqqt3/assignment_loss‚0.9047926549523414 1loss/SRqqt3/detection_loss‚0.9563032055560213 /loss/FBt2/detection_loss‚1.9121912370125453 epoch‚8 /loss/FRt2/assignment_loss‚3.796801493219707 3loss/FBt4/assignment_loss‚0.0022518533223774284 .loss/FRt1/detection_loss‚2.956836054402013 2loss/SRqqt2/assignment_loss‚1.1504304995439176 .loss/FBt4/detection_loss‚4.785903781652451 trainer/global_step‚10699 _step‚879 /loss/FRt4/assignment_loss‚3.159470170736313 /loss/FBt2/assignment_loss‚0.065043537498253 2loss/FBt3/assignment_loss‚0.008081294921708258 /loss/FRt2/detection_loss‚2.2553633517344007 0loss/SRqqt2/detection_loss‚2.206316593392142 0loss/SRqqt4/detection_loss‚2.525214200670069 /loss/FBt3/detection_loss‚0.8914230161894774 0loss/FRt3/assignment_loss‚1.5035169422626495 _runtime‚12626.362211407x÷M©bÖ+Ü 2loss/SRqqt2/assignment_loss‚1.1504304995439176 .loss/FBt4/detection_loss‚4.785903781652451 trainer/global_step‚10699 _step‚879 /loss/FRt4/assignment_loss‚3.159470170736313 /loss/FBt2/assignment_loss‚0.065043537498253 2loss/FBt3/assignment_loss‚0.008081294921708258 /loss/FRt2/detection_loss‚2.2553633517344007 0loss/SRqqt2/detection_loss‚2.206316593392142 0loss/SRqqt4/detection_loss‚2.525214200670069 /loss/FBt3/detection_loss‚0.8914230161894774 0loss/FRt3/assignment_loss‚1.5035169422626495 _runtime‚12626.362211407 2loss/FBt1/assignment_loss‚0.023536896591085425 .loss/FRt3/detection_loss‚1.387158755833904 /loss/FBt1/detection_loss‚2.5076894793245526 %loss/total_loss‚43.01349350384428 2loss/SRqqt4/assignment_loss‚1.2784093455834822 .loss/FRt4/detection_loss‚2.718955032135311 0loss/SRqqt1/detection_loss‚1.778451959292094 % _timestamp‚1.7884765731772368e+09 0loss/FRt1/assignment_loss‚3.0012282440739293 1loss/SRqqt1/assignment_loss‚1.238123903943536 2loss/SRqqt3/assignment_loss‚0.9047926549523414 1loss/SRqqt3/detection_loss‚0.9563032055560213 /loss/FBt2/detection_loss‚1.9121912370125453 epoch‚8 /loss/FRt2/assignment_loss‚3.796801493219707 3loss/FBt4/assignment_loss‚0.0022518533223774284 .loss/FRt1/detection_loss‚2.956836054402013ïÝžw3‘×+‚(:†  ùçÔäÊ¿ˆ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚115.966' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.9915 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ{KâØ+‚(:ö  ùçÔ€š×ˆ) disk./.usagePercent‚83.18135686532034# disk.nvme1n1.out‚71895.11328125! disk.sdb2.out‚1393.6884765625 disk.rbd0.out‚ 3.3671875) proc.memory.availableMB‚ 9847.76171875# proc.memory.rssMB‚ 2953.62109375 cpu‚25.190661179168593 disk.md0p1.in‚ 10.98046875 disk.md0p1.out‚ 1343.0546875 disk.md0.in‚ 10.98046875 disk.sda.out‚1393.6884765625! disk.dm-0.out‚149181.30078125 disk.rbd0.in‚0.015625$ memory_percent‚39.89403247833252 proc.cpu.threads‚17 disk.sdb.out‚1393.6884765625# disk.nvme0n1.in‚329984.59765625 disk.sda1.in‚0 network.sent‚ 2.8673081e+07" disk.nvme1n1.in‚201429.0234375 disk.sdb1.out‚0 disk.sda1.out‚0* proc.memory.percent‚18.027472496032715 disk.md0.out‚ 1343.0546875 disk.sdb2.in‚5.40625 network.recv‚ 4.352257e+06$ disk./.usageGB‚2974.968231201172 disk.rbd2.in‚0 disk.rbd2.out‚ 3.3671875 disk.sdb.in‚5.40625 disk.sda2.in‚ 5.57421875! disk.sda2.out‚1393.6884765625 disk.sdb1.in‚0 disk.sda.in‚ 5.57421875 disk.nvme0n1.out‚ 77286.1875 disk.dm-0.in‚531413.62109375ùG”»›Ù+‚(: ¯ùçÔâˆò‡ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚110.681' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚27.670250000000003 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^HOý…Ú+‚(:ú ¯ùçԮ㈠disk.rbd0.out‚ 3.3671875# disk.nvme0n1.in‚329984.59765625 disk.rbd0.in‚0.015625% memory_percent‚39.891719818115234) disk./.usagePercent‚83.18833225080003! disk.sdb2.out‚1395.9091796875" disk.nvme1n1.in‚201429.0234375 disk.sdb2.in‚5.40625 disk.md0.out‚ 1345.21484375 network.recv‚ 4.358561e+06 disk.sdb.out‚1395.9091796875 disk.sda1.in‚0 disk.sda2.in‚ 5.57421875# disk.nvme1n1.out‚72087.94921875 disk.sdb1.out‚0 disk.dm-0.in‚531413.62109375! disk.dm-0.out‚149518.36328125 disk.sda1.out‚0 disk.md0p1.in‚ 10.98046875 disk.md0.in‚ 10.98046875# proc.memory.rssMB‚ 2953.62109375 disk.sdb.in‚5.40625 disk.sdb1.in‚0 disk.sda.in‚ 5.57421875! disk.sda2.out‚1395.9091796875 disk.md0p1.out‚ 1345.21484375 disk.rbd2.in‚0 disk.rbd2.out‚ 3.3671875' proc.memory.availableMB‚ 9848.140625$ disk./.usageGB‚2975.217704772949 disk.sda.out‚1395.9091796875* proc.memory.percent‚18.027472496032715 network.sent‚ 2.8767187e+07" disk.nvme0n1.out‚ 77430.4140625 cpu‚25.190474650039796 proc.cpu.threads‚17&lIƒÛ+‚(:ø ¾ùçÔÄÈ¢‡" disk.nvme0n1.out‚ 77502.5703125 disk.dm-0.in‚531413.62109375 disk.md0p1.out‚ 1346.8125 disk.rbd2.in‚0 disk.sda.in‚ 5.57421875% disk./.usageGB‚2975.0926780700684# disk.nvme0n1.in‚329984.59765625" disk.sda2.out‚1397.57275390625 disk.md0p1.in‚ 10.98046875 disk.sdb2.in‚5.40625" disk.nvme1n1.in‚201429.0234375# disk.nvme1n1.out‚72145.70703125 disk.sdb1.out‚0) disk./.usagePercent‚83.18483645185975! disk.dm-0.out‚149648.27734375 disk.rbd0.in‚0.015625 disk.sdb1.in‚0 disk.sda1.in‚0" disk.sdb2.out‚1397.57275390625 disk.md0.in‚ 10.98046875 disk.rbd2.out‚ 3.3671875 disk.sdb.in‚5.40625! disk.sdb.out‚1397.57275390625 proc.cpu.threads‚17 network.sent‚ 2.8775928e+07 disk.sda1.out‚0 disk.md0.out‚ 1346.8125$ memory_percent‚39.89236354827881) proc.memory.availableMB‚ 9848.03515625 disk.rbd0.out‚ 3.3671875# proc.memory.rssMB‚ 2953.62109375* proc.memory.percent‚18.027472496032715 network.recv‚ 4.363294e+06! disk.sda.out‚1397.57275390625 cpu‚25.190255811427857 disk.sda2.in‚ 5.57421875 9f£Ü+‚(:… ¾ùçÔë…ʇ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚100.482' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.1205 gpu.0.smClock‚1170 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º4CÌÝ+‚(:… ÍùçÔŽÞë‡ gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚134.908' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.727 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:ê΃ŠÞ+‚(:ÿ ÍùçÔǹɈ disk.sda2.in‚ 5.57421875 disk.md0p1.in‚ 10.98046875% memory_percent‚39.893150329589844 disk.sda1.out‚0* proc.memory.percent‚18.027472496032715 network.sent‚ 2.8781798e+07# disk.nvme0n1.out‚77630.76171875& proc.memory.availableMB‚ 9847.90625 cpu‚25.190084857236787" disk.sda2.out‚1399.06982421875) disk./.usagePercent‚83.18483645185975" disk.sdb2.out‚1399.06982421875 disk.md0.out‚ 1348.24609375" disk.nvme1n1.in‚201429.0234375" disk.nvme1n1.out‚ 72298.5078125 disk.sdb1.out‚0# disk.nvme0n1.in‚329984.59765625 network.recv‚ 4.367562e+06 disk.md0.in‚ 10.98046875 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 3.37890625 disk.sdb.in‚5.40625! disk.sdb.out‚1399.06982421875! disk.dm-0.out‚149929.26953125 disk.sda1.in‚0% disk./.usageGB‚2975.0926780700684! disk.sda.out‚1399.06982421875 disk.dm-0.in‚531413.62109375# proc.memory.rssMB‚ 2953.62109375 proc.cpu.threads‚17 disk.sdb2.in‚5.40625 disk.rbd2.in‚0 disk.sda.in‚ 5.57421875 disk.md0p1.out‚ 1348.24609375 disk.rbd2.out‚ 3.3828125 disk.sdb1.in‚07ŽÊ•4ß+/ _wandbruntime‚12686  _runtime‚12686ÂV1Ëà+Å ' lr-AdamW/pg1‚3.0513593339876506e-05 ' lr-AdamW/pg2‚3.0513593339876506e-05 trainer/global_step‚10749 % _timestamp‚1.7884766328564298e+09 _runtime‚12686.041375976 _step‚880³ÍjÐá+Ê _runtime‚12686.041375976 _step‚880 ' lr-AdamW/pg1‚3.0513593339876506e-05 ' lr-AdamW/pg2‚3.0513593339876506e-05 trainer/global_step‚10749 % _timestamp‚1.7884766328564298e+09ðº>³4â+/ _wandbruntime‚12686  _runtime‚12686«=²zã+ìO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚458849997254144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 204221269 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 292152624 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 292152624 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 292178968 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚707 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 292178968 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 204221269 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21354499635712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 496400237 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚463422125231616 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚437640732383746 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚416346994664368 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚10749 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21354499635712 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 122017946 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 122017948 BDevicey<4GxStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 199587892 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚437495749289984 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚458849997254144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚458850248925696 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚463422101291520 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 % _timestamp‚1.7884766328574305e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚437495749289984 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 199587185 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚77569944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21354265967104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21293966700462 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21354265967104 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 199587892 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚437495749289984 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚458849997254144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚458850248925696 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚463422101291520 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 % _timestamp‚1.7884766328574305e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚437495749289984 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 199587185 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚77569944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21354265967104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21293966700462 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21354265967104 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚458849997254144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 204221269 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 292152624 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 292152624 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 292178968 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚707 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 292178968 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 204221269 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21354499635712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 496400237 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚463422125231616 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚437640732383746ñ€î˜4å+/ _wandbruntime‚12686  _runtime‚12686¹å4ü&æ+öM ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21356463674880 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚687 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚463463785261568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21356229992960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21356463674880 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 292205686 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 496445230 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 496445230 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚437535017324032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚689 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚437680221230156 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚416384525021792 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 199605298 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7884766338055818e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 496418858 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 204239542 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 204239544 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚882 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 292205686 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚437535035326976 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 122028750 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21295925189656 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚458891247316992 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 292179316 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚77577237 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21295696208364 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚437535035326976 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚416384542061152 trainer/global_step‚10749 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 292179316 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 199605987 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21716728448000 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚437535017324032 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 204239542 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚458891499001856 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 496418858 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21356229992960 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚458891499001856 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21717019054080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 204239544 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 122028748 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚463463809201664¶¥&¬>ç+ûM YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21295696208364 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚437535035326976 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚416384542061152 trainer/global_step‚10749 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 292179316 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 199605987 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21716728448000 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚437535017324032 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 204239542 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚458891499001856 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 496418858 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21356229992960 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚458891499001856 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21717019054080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 204239544 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 122028748 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚463463809201664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21356463674880 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚687 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚463463785261568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21356229992960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21356463674880 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 292205686 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 496445230 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 496445230 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚437535017324032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚689 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚437680221230156 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚416384525021792 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 199605298 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7884766338055818e+09 D6DeviceStatsMonitor.on_train_batch_end/active.a½ Pà ll.freed‚ 496418858 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 204239542 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 204239544 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚882 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 292205686 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚437535035326976 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 122028750 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21295925189656 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚458891247316992 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 292179316 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚77577237ò\ý°4è+/ _wandbruntime‚12686  _runtime‚12686oreìaé+Û 1loss/SRqqt3/detection_loss‚0.9299281092050175 .loss/FBt1/detection_loss‚2.395624437115409 /loss/FBt4/detection_loss‚1.9632050144044977 &loss/total_loss‚41.059885695479686 trainer/global_step‚10749 1loss/FBt4/assignment_loss‚0.36945427383149143 .loss/FRt4/detection_loss‚3.033878197669983 /loss/FBt2/detection_loss‚2.8253503665328026 /loss/FBt3/detection_loss‚0.5233318575808559 epoch‚8 $ _timestamp‚1.788476633808063e+09 _runtime‚12686.99276213 _step‚883 2loss/SRqqt1/assignment_loss‚1.5188766060079015 1loss/SRqqt4/detection_loss‚2.5733641173158373 2loss/SRqqt2/assignment_loss‚1.6040523952016463 2loss/SRqqt4/assignment_loss‚1.6208061780780554 4loss/FBt2/assignment_loss‚0.00015993141504466735 1loss/FBt1/assignment_loss‚0.20215859741936335 /loss/FRt2/detection_loss‚2.2159886130780735 /loss/FRt2/assignment_loss‚2.980328752639446 2loss/SRqqt3/assignment_loss‚0.8412939599608459 .loss/FRt1/detection_loss‚2.672622413248629 /loss/FRt3/detection_loss‚1.4583555991731343 1loss/SRqqt2/detection_loss‚2.5388010843442035 /loss/FRt1/assignment_loss‚2.764721673888129 1loss/SRqqt1/detection_loss‚2.0807087937506235 0loss/FRt3/assignment_loss‚1.0398996462745052 1loss/FBt3/assignment_loss‚0.12788964386106455 /loss/FRt4/assignment_loss‚2.779085433483124,®/fê+à 1loss/FBt4/assignment_loss‚0.36945427383149143 .loss/FRt4/detection_loss‚3.033878197669983 /loss/FBt2/detection_loss‚2.8253503665328026 /loss/FBt3/detection_loss‚0.5233318575808559 epoch‚8 $ _timestamp‚1.788476633808063e+09 _runtime‚12686.99276213 _step‚883 2loss/SRqqt1/assignment_loss‚1.5188766060079015 1loss/SRqqt4/detection_loss‚2.5733641173158373 2loss/SRqqt2/assignment_loss‚1.6040523952016463 2loss/SRqqt4/assignment_loss‚1.6208061780780554 4loss/FBt2/assignment_loss‚0.00015993141504466735 1loss/FBt1/assignment_loss‚0.20215859741936335 /loss/FRt2/detection_loss‚2.2159886130780735 /loss/FRt2/assignment_loss‚2.980328752639446 2loss/SRqqt3/assignment_loss‚0.8412939599608459 .loss/FRt1/detection_loss‚2.672622413248629 /loss/FRt3/detection_loss‚1.4583555991731343 1loss/SRqqt2/detection_loss‚2.5388010843442035 /loss/FRt1/assignment_loss‚2.764721673888129 1loss/SRqqt1/detection_loss‚2.0807087937506235 0loss/FRt3/assignment_loss‚1.0398996462745052 1loss/FBt3/assignment_loss‚0.12788964386106455 /loss/FRt4/assignment_loss‚2.779085433483124 1loss/SRqqt3/detection_loss‚0.9299281092050175 .loss/FBt1/detection_loss‚2.395624437115409 /loss/FBt4/detection_loss‚1.9632050144044977 &loss/total_loss‚41.059885695479686 trainer/global_step‚10749ó9ŠGk‚ë+‚(:÷ ÜùçÔÚîë‰! disk.sda2.out‚1400.8798828125 network.sent‚ 2.8798378e+07) disk./.usagePercent‚83.17978287584782 disk.rbd2.out‚ 3.4453125 disk.sdb1.in‚0 disk.rbd0.in‚0.015625 disk.sdb.out‚1400.8798828125' proc.memory.availableMB‚ 9848.078125 disk.sdb.in‚5.40625 disk.md0.out‚ 1349.99609375# proc.memory.rssMB‚ 2953.62109375 disk.rbd0.out‚ 3.37890625! disk.dm-0.out‚150072.44921875* proc.memory.percent‚18.027472496032715$ disk./.usageGB‚2974.911937713623# disk.nvme1n1.out‚72425.66796875 disk.sdb1.out‚0 disk.sda.out‚1400.8798828125 disk.dm-0.in‚531413.62109375 disk.sda1.out‚0! disk.sdb2.out‚1400.8798828125 cpu‚25.189894793563617 disk.rbd2.in‚0 disk.sda.in‚ 5.57421875 disk.nvme0n1.out‚ 77646.78125 disk.sda1.in‚0 disk.sdb2.in‚5.40625 disk.md0.in‚ 10.98046875" disk.nvme1n1.in‚201429.0234375 disk.sda2.in‚ 5.57421875 disk.md0p1.in‚ 10.98046875# memory_percent‚39.8921012878418 proc.cpu.threads‚17 network.recv‚ 4.371763e+06# disk.nvme0n1.in‚329984.59765625 disk.md0p1.out‚ 1349.99609375ሆœì+‚(:‘ ÜùçÔáù‰ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Óš„Zœí+‚(:‘ ëùçÔ®¹å‡ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µkˆî+‚(:ý ëùçÔª×ý‡ disk.dm-0.in‚531413.62109375! disk.dm-0.out‚150246.12890625 disk.rbd0.in‚0.015625# disk.nvme1n1.out‚72477.95703125! disk.sdb.out‚1402.87353515625! disk.sda.out‚1402.87353515625 disk.sda2.in‚ 5.57421875 disk.md0.in‚ 10.98046875 disk.rbd2.out‚ 3.4453125# memory_percent‚39.8921012878418' proc.memory.availableMB‚ 9848.078125 cpu‚25.189678909090524) disk./.usagePercent‚83.15328306664217! disk.nvme0n1.out‚ 77768.171875 disk.sda1.in‚0# proc.memory.rssMB‚ 2953.62109375 network.recv‚ 4.378101e+06% disk./.usageGB‚2973.9641761779785 disk.sda.in‚ 5.57421875 disk.sda1.out‚0 proc.cpu.threads‚17 network.sent‚ 2.8892504e+07" disk.sda2.out‚1402.87353515625 disk.rbd0.out‚ 3.44140625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0p1.in‚ 10.98046875 disk.sdb2.in‚5.40625" disk.sdb2.out‚1402.87353515625" disk.nvme1n1.in‚201429.0234375 disk.sdb.in‚5.40625* proc.memory.percent‚18.027472496032715 disk.md0p1.out‚ 1351.93359375 disk.md0.out‚ 1351.93359375 disk.rbd2.in‚0# disk.nvme0n1.in‚329984.59765625^ß?f‘ï+‚(:† úùçÔó¹Á‡ gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚140.478' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.1195 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“( ê{ð+‚(:ð úùçÔÚÇ̇ disk.sda.out‚1404.548828125 disk.sda2.out‚1404.548828125 disk.sdb2.in‚5.40625 disk.md0.out‚ 1353.53125" disk.nvme1n1.in‚201429.0234375 network.sent‚ 2.8898925e+07 disk.dm-0.in‚531413.62109375! disk.dm-0.out‚150289.10546875 disk.sdb2.out‚1404.548828125 disk.md0.in‚ 10.98046875 disk.sdb1.in‚0 disk.sda.in‚ 5.57421875 network.recv‚ 4.382927e+06 disk.rbd2.in‚0 disk.sda2.in‚ 5.57421875 disk.md0p1.in‚ 10.98046875 disk.rbd2.out‚ 3.53271484375 disk.sda1.in‚0 disk.md0p1.out‚ 1353.53125 disk.rbd0.out‚ 3.44140625# proc.memory.rssMB‚ 2953.62109375* proc.memory.percent‚18.027472496032715% disk./.usageGB‚2973.7560806274414 disk.sda1.out‚0# disk.nvme0n1.out‚77768.23046875# memory_percent‚39.8951530456543 proc.cpu.threads‚17) disk./.usagePercent‚83.14746462795442 disk.rbd0.in‚0.015625# disk.nvme0n1.in‚329984.59765625' proc.memory.availableMB‚ 9847.578125 cpu‚25.189476997089493 disk.nvme1n1.out‚ 72520.875 disk.sdb.in‚5.40625 disk.sdb.out‚1404.548828125 disk.sdb1.out‚0õI‡ñ+‚(:ü ‰úçÔ…€Ã‡ disk.sdb.out‚1406.0986328125 disk.sdb1.in‚0# disk.nvme0n1.in‚329984.59765625) proc.memory.availableMB‚ 9847.45703125* proc.memory.percent‚18.027472496032715 disk.md0p1.in‚ 10.98046875 disk.md0.out‚ 1355.01171875 disk.rbd2.in‚0 disk.sda.out‚1406.0986328125 disk.dm-0.out‚ 150290.1875 proc.cpu.threads‚17 network.recv‚ 4.387094e+06" disk.nvme1n1.out‚ 72521.6484375( disk./.usagePercent‚83.1481140839345$ disk./.usageGB‚2973.779308319092 disk.rbd2.out‚ 3.53271484375 disk.sda.in‚ 5.57421875 disk.sda1.in‚0! disk.sdb2.out‚1406.0986328125 disk.rbd0.out‚ 3.5205078125 disk.sdb1.out‚0" disk.nvme0n1.out‚ 77768.5390625 disk.sda1.out‚0 disk.sdb2.in‚5.40625" disk.nvme1n1.in‚201429.0234375 disk.dm-0.in‚531413.62109375! disk.sda2.out‚1406.0986328125$ memory_percent‚39.89589214324951 cpu‚25.189293862947764 disk.md0.in‚ 10.98046875# proc.memory.rssMB‚ 2953.62109375 disk.md0p1.out‚ 1355.01171875 disk.rbd0.in‚0.015625 disk.sdb.in‚5.40625 disk.sda2.in‚ 5.57421875 network.sent‚ 2.8906995e+07ÁÞ&%’ò+‚(:‡ ‰úçÔ︚ gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚141.087' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.27175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º>4ó+/ _wandbruntime‚12736  _runtime‚12736†ÐãÉô+à _step‚884 & lr-AdamW/pg1‚2.830650626087466e-05 & lr-AdamW/pg2‚2.830650626087466e-05 trainer/global_step‚10799 % _timestamp‚1.7884766836926637e+09 _runtime‚12736.877645923ýš¹ÿÎõ+È _step‚884 & lr-AdamW/pg1‚2.830650626087466e-05 & lr-AdamW/pg2‚2.830650626087466e-05 trainer/global_step‚10799 % _timestamp‚1.7884766836926637e+09 _runtime‚12736.877645923ôç/ j4ö+/ _wandbruntime‚12736  _runtime‚12736‹UöÒò'÷+ìO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 293490576 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚460912766481920 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚439460051139584 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21814619743232 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚439460033136640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚77921403 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚487321244582400 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚77922092 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚439460033136640 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚465506323734016 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚418223529574928 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21391904740970 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 205135019 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 498651939 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚465506299793920 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚418223512535568 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚689 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚460912514810368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _step‚885 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 293516920 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚439460051139584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 205135017 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21814920848384 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21452715342336 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325045248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚12736.878469067 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 122558048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21452715342336 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21452481673728 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚439615188295454 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 4986255938täl'ø+ñO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7884766836933985e+09 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚460912514810368 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 205135019 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 498651939 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚465506299793920 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚418223512535568 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚689 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚460912514810368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _step‚885 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 293516920 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚439460051139584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 205135017 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21814920848384 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21452715342336 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325045248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚12736.878469067 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 122558048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21452715342336 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21452481673728 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚439615188295454 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 498625593 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 293490576 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚460912766481920 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚439460051139584 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21814619743232 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚439460033136640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚77921403 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚487321244582400 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚77922092 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚439460033136640 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚465506323734016 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚418223529574928 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21391904740970 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 122568848 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚460953764873216 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21393863230164 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚460954016558080 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚439499337176576 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21454445699584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚77928000 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 498670560 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21816907252224 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 498696932 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21454445699584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚662 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 293517268 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚439499337176576 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884766846566556e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 200497512 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚418261042892992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 293517268 _runtime‚12737.84223514 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 200496848 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 122568850 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 498670560 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 205153294 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 498696932 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚439499319173632 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚465547983763968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 293543638 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21454679381504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0žLF–'û+úM ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 498696932 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚439499319173632 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚465547983763968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 293543638 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21454679381504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 122568848 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚460953764873216 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21393863230164 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚460954016558080 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚439499337176576 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21454445699584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚77928000 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 498670560 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21816907252224 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 498696932 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21454445699584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚662 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 293517268 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚439499337176576 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884766846566556e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 200497512 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚418261042892992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 293517268 _runtime‚12737.84223514 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 200496848 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 122568850 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 498670560 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 205153294 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚664öUþ³*4ü+/ _wandbruntime‚12737  _runtime‚12737ÑI[ý+Õ _step‚887 0loss/FRt2/assignment_loss‚3.1287153718563228 0loss/FBt2/assignment_loss‚0.1211360775162645 /loss/FRt4/detection_loss‚2.6537777802773883 /loss/FBt2/detection_loss‚2.0719202756881714 %loss/total_loss‚40.38775230125666 /loss/FRt4/assignment_loss‚2.726375183888844 2loss/SRqqt1/assignment_loss‚1.6305902160895176 0loss/FBt4/assignment_loss‚0.3041305831982726 1loss/SRqqt1/detection_loss‚1.8972678024035234 /loss/FBt3/detection_loss‚0.3578281399526008 .loss/FBt4/detection_loss‚2.809242230653763 _runtime‚12737.843818587 2loss/SRqqt3/assignment_loss‚0.9871572173104378 1loss/SRqqt4/assignment_loss‚1.458248978273736 /loss/FRt3/detection_loss‚1.5896620081182111 1loss/SRqqt2/detection_loss‚2.3858171738684177 1loss/SRqqt4/detection_loss‚2.2953699076617204 % _timestamp‚1.7884766846591208e+09 /loss/FRt2/detection_loss‚1.8247355994960974 /loss/FBt1/detection_loss‚2.6131254017353056 epoch‚8 .loss/FRt1/assignment_loss‚2.56465701200068 0loss/FRt3/assignment_loss‚0.9528190796835381 1loss/SRqqt2/assignment_loss‚2.074779972434044 0loss/FBt1/assignment_loss‚0.0515423215690987 1loss/FBt3/assignment_loss‚0.08043898875517261 .loss/FRt1/detection_loss‚2.887231769040227 trainer/global_step‚10799 1loss/SRqqt3/detection_loss‚0.9211832097853008“'A>`þ+Ú _step‚887 0loss/FRt2/assignment_loss‚3.1287153718563228 0loss/FBt2/assignment_loss‚0.1211360775162645 /loss/FRt4/detection_loss‚2.6537777802773883 /loss/FBt2/detection_loss‚2.0719202756881714 %loss/total_loss‚40.38775230125666 /loss/FRt4/assignment_loss‚2.726375183888844 2loss/SRqqt1/assignment_loss‚1.6305902160895176 0loss/FBt4/assignment_loss‚0.3041305831982726 1loss/SRqqt1/detection_loss‚1.8972678024035234 /loss/FBt3/detection_loss‚0.3578281399526008 .loss/FBt4/detection_loss‚2.809242230653763 _runtime‚12737.843818587 2loss/SRqqt3/assignment_loss‚0.9871572173104378 1loss/SRqqt4/assignment_loss‚1.458248978273736 /loss/FRt3/detection_loss‚1.5896620081182111 1loss/SRqqt2/detection_loss‚2.3858171738684177 1loss/SRqqt4/detection_loss‚2.2953699076617204 % _timestamp‚1.7884766846591208e+09 /loss/FRt2/detection_loss‚1.8247355994960974 /loss/FBt1/detection_loss‚2.6131254017353056 epoch‚8 .loss/FRt1/assignment_loss‚2.56465701200068 0loss/FRt3/assignment_loss‚0.9528190796835381 1loss/SRqqt2/assignment_loss‚2.074779972434044 0loss/FBt1/assignment_loss‚0.0515423215690987 1loss/FBt3/assignment_loss‚0.08043898875517261 .loss/FRt1/detection_loss‚2.887231769040227 trainer/global_step‚10799 1loss/SRqqt3/detection_loss‚0.9211832097853008÷Ö冃Œÿ+‚(:  ˜úçÔùÞ‹Š network.sent‚ 2.8910937e+07) disk./.usagePercent‚83.14904363047194 disk.rbd2.out‚ 3.595703125 disk.sdb1.out‚0" disk.sda2.out‚1408.04443359375% disk./.usageGB‚2973.8125534057617" disk.nvme1n1.out‚ 72590.3828125 disk.sdb.in‚5.40625 disk.sda1.in‚0# disk.nvme0n1.in‚329993.81640625 disk.sda1.out‚0 disk.md0p1.in‚ 10.98046875 disk.rbd0.out‚ 3.5205078125 disk.dm-0.in‚ 531424.53125$ memory_percent‚39.89231586456299) proc.memory.availableMB‚ 9848.04296875 disk.sdb2.in‚5.40625 disk.rbd0.in‚0.015625 disk.rbd2.in‚0# disk.nvme1n1.in‚201430.71484375! disk.sdb.out‚1408.04443359375! disk.sda.out‚1408.04443359375 disk.sda2.in‚ 5.57421875# proc.memory.rssMB‚ 2953.62109375 network.recv‚ 4.391375e+06 cpu‚25.189119867562287 proc.cpu.threads‚17" disk.sdb2.out‚1408.04443359375 disk.md0.in‚ 10.98046875 disk.md0.out‚ 1356.8984375 disk.sda.in‚ 5.57421875! disk.dm-0.out‚150359.10546875 disk.sdb1.in‚0# disk.nvme0n1.out‚77768.72265625 disk.md0p1.out‚ 1356.8984375* proc.memory.percent‚18.027472496032715Ûÿ”7€,‚(:‚ ˜úçÔôÄóˆ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚94.58' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚23.645 gpu.0.smClock‚1245 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~ÇQÚ,‚(:… §úçÔÕØì… gpu.0.gpu‚43.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚139.528' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 w=‚,‚(:„  §úçÔƦå†$ disk./.usageGB‚2973.781021118164 disk.sda.in‚ 5.57421875! disk.sda.out‚1409.60986328125 disk.md0.out‚ 1358.41015625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0p1.out‚ 1358.41015625# disk.nvme0n1.in‚330003.13671875 disk.sda1.in‚0# disk.nvme1n1.out‚72669.01171875) proc.memory.availableMB‚ 9847.79296875# proc.memory.rssMB‚ 2953.62109375 cpu‚25.188931297121883 network.sent‚ 2.9018127e+07 disk.dm-0.out‚ 150439.296875 disk.sda2.in‚ 5.57421875 disk.md0p1.in‚ 10.98046875" disk.sdb2.out‚1409.60986328125 disk.md0.in‚ 10.98046875$ memory_percent‚39.89384174346924 proc.cpu.threads‚17) disk./.usagePercent‚83.14816197451343 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 3.58349609375# disk.nvme0n1.out‚77770.28515625" disk.sda2.out‚1409.60986328125 disk.rbd2.in‚0# disk.nvme1n1.in‚201430.71484375! disk.sdb.out‚1409.60986328125* proc.memory.percent‚18.027472496032715 disk.rbd2.out‚ 3.595703125 network.recv‚ 4.398112e+06 disk.dm-0.in‚531433.8515625 disk.sda1.out‚0 disk.sdb2.in‚5.40625 disk.sdb.in‚5.40625FwL³ƒ,‚(:‚  ¶úçÔ…‘³‡ network.sent‚ 2.9024503e+07 disk.sda1.out‚0% disk./.usageGB‚2974.0874404907227 disk.rbd2.out‚ 3.65869140625 disk.sdb1.out‚0# disk.nvme0n1.in‚330003.13671875 disk.sda2.in‚ 5.57421875 disk.md0p1.out‚ 1360.01953125 disk.rbd0.out‚ 3.58349609375# disk.nvme1n1.in‚201430.71484375 disk.nvme0n1.out‚ 77770.40625 proc.cpu.threads‚17 disk.md0.in‚ 10.98046875 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.dm-0.in‚531433.8515625 disk.md0p1.in‚ 10.98046875) disk./.usagePercent‚83.15672958841664 disk.sdb2.in‚5.40625! disk.sdb2.out‚1411.2841796875* proc.memory.percent‚18.027472496032715! disk.sda2.out‚1411.2841796875 network.recv‚ 4.402595e+06 disk.sdb.out‚1411.2841796875 disk.sdb1.in‚0 disk.sda1.in‚0$ memory_percent‚39.89684581756592) proc.memory.availableMB‚ 9847.30078125 disk.sdb.in‚5.40625 disk.sda.in‚ 5.57421875 disk.sda.out‚1411.2841796875! disk.dm-0.out‚150659.22265625# proc.memory.rssMB‚ 2953.62109375 disk.md0.out‚ 1360.01953125# disk.nvme1n1.out‚72888.81640625 cpu‚25.188727206123094;”¼¼‘„,‚(:† ¶úçÔÝßÙ‡ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.357' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.83925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^Þ Þ4…,/ _wandbruntime‚12790  _runtime‚12790Ò YÇʆ,Ä ' lr-AdamW/pg2‚2.6179427502098677e-05 trainer/global_step‚10849 $ _timestamp‚1.788476737563968e+09 _runtime‚12790.749096879 _step‚888 ' lr-AdamW/pg1‚2.6179427502098677e-05eÞV–χ,É $ _timestamp‚1.788476737563968e+09 _runtime‚12790.749096879 _step‚888 ' lr-AdamW/pg1‚2.6179427502098677e-05 ' lr-AdamW/pg2‚2.6179427502098677e-05 trainer/global_step‚10849øÞllm4ˆ,/ _wandbruntime‚12790  _runtime‚12790Öýñu‚‰,ìO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚10849 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21550697380352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚441424352989184 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21550931048960 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 123098148 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 206048769 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚441424334986240 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 294828528 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚489505458846208 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884767375652206e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚462975032366592 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚467590522236416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 _runtime‚12790.750462406 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚441589890227606 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚420100047446128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 123098146 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚462975032366592 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.=®—dpfreed‚489505148481024 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 294828528 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚441424352989184 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 201366962 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚462975284038144 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21550697380352 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21489613800394 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 500903641 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 294854872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚695 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21914936609792n©±Å÷'Š,ñO JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 201366962 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚462975284038144 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21550697380352 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21489613800394 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 500903641 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 294854872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚695 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21914936609792 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚10849 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21550697380352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚441424352989184 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21550931048960 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 123098148 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 206048769 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚441424334986240 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 294828528 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚489505458846208 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884767375652206e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚462975032366592 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚467590522236416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 _runtime‚12790.750462406 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚441589890227606 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚420100047446128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 123098146 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚462975032366592 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚489505148481024 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 294828528 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚441424352989184 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 201384468 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 206067044 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21552895088128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚12791.822861021 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21491801270672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21552661406208 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 500922262 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚420137577803552 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚463016534114304 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 123108950 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 500922262 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚441629379074224 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 500948634 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 294881590 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 123108948 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚467632182266368 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21491572289380 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚420137560764192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚10849 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 206067042 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 294881590 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚441463621023232 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚660 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚441629133053572 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚463016282429440 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 206067042 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 294855220 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚463016534114304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚489549137592832 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 304060416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 500948634 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21916931386368 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21552895088128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _step‚890 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 201385128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚658 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280120320 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚441463621023232 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚78275520 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21552661406208 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496;B<ÆF,ûM K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 294855220 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚441463639026176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚489548833532416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7884767386371527e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚441463621023232 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚78275520 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21552661406208 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 206067044 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 201384468 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 206067044 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21552895088128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚12791.822861021 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21491801270672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21552661406208 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 500922262 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚420137577803552 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚463016534114304 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 123108950 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 500922262 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚441629379074224 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 500948634 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 294881590 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 123108948 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚467632182266368 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21491572289380 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚420137560764192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚10849 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 206067042 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 294881590 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚441463621023232 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚660 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_ba1%"5» tch_end/requested_bytes.all.freed‚441629133053572 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚463016282429440 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 206067042 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 294855220 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚463016534114304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚489549137592832 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 304060416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 500948634 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21916931386368 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21552895088128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _step‚890 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 201385128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚658 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280120320 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6966900736 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 202781811 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚582 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚465791550366208 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 503512823 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 156409344 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚470412111264256 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚379 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6741059457 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 572522496 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚465791744181248 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚21637019970560 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚465791550366208 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 193815040 % _timestamp‚1.7884768327269094e+09 ^PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚ 124016230 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193815040 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚273 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚22002074414080 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚309 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚21637019970560 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 296146249 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚470412135204352 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚582 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 503512823 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚444154724210688 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6561988608 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 503533536 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚492414185678336 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚422553441715664 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚21575674602310 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 7134511104 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚36 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20711 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚22002206883328 _runtime‚12885.912657904 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚309 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚21636844158464 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 207387287 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚377 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 132469248 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 503533536 _step‚892;`Ú¡,ØT TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1300 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚444128927458142 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚309 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚78765581 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188859832 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 296125538 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚444129116317974 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20713 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6966900736 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚422553424676304 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚273 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 207387285 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚444154724210688 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚78765204 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚582 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3532 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6458321920 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚21575502781838 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6966900736 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 202781811 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚582 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚465791550366208 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 503512823 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 156409344 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚470412111264256 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚379 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6741059457 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 572522496 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚465791744181248 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚21637019970560 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚465791550366208 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 193815040 % _timestamp‚1.7884768327269094e+09 ^PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚ 124016230 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193815040 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚273 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚22002074414080 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚309 dQDeviceStÙlTÁatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚21637019970560 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 296146249 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚470412135204352 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚582 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 503512823 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚444154724210688 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6561988608 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 503533536 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚492414185678336 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚422553441715664 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚21575674602310 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 7134511104 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚36 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20711 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚22002206883328 _runtime‚12885.912657904 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚309 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚21636844158464 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 207387287 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚377 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 132469248 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 503533536 _step‚892 FDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 202799018 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1300 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚21576146810122 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚582 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚444184771881984 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚309 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 296155157 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚21637318018560 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚309 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 193815040 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚465822283715584 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 156409344 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚422579001365296 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚582 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚309 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚470442496957952 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚422578984325936 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 193815040 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 207402667 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 347418624 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 _runtime‚12887.053383424 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚470442473017856§îzÜh)¤,âR JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚465822089900544 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1319 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚465822283715584 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32006 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚492445177701376 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6741059457 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚582 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188859832 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚273 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 207402669 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3532 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 296134446 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚378 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚21637493830656 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 175812096 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 296155157 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚22002680743424 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 560288777 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 % _timestamp‚1.7884768338679113e+09 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 572522496 \NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚ 124027912 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 503557826 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 565025792 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚21637493830656 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20713 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 202798638 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 207402667 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20711 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 132469248 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚309 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 193815040 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚465822283715584 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 156409344 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚422579001365296 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚582 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚309 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚470442496957952 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚422578984325936 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 193815040 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 207402667 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 347418624 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 _runtime‚12887.053383424 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚470442473017856 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚444155148175418 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚21575974989650 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3532 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚444154959315586 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 7134511104 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 trainer/global_step‚10880 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚36 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚273 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 503537113 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 175812096 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 _step‚893 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6966900736 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚444184789884928 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 503537113 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚78771106 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 207402669 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚465822089900544 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 7134511104 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 503557826 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 202799018 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1300 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚21576146810122 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚582 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚444184771881984 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚309 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 296155157 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚21637318018560 B&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.17929291725158691 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.27369406819343567 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5693391561508179 particle/accuracy_2_of_5‚1 +particle/specificity‚0.9506559371948242 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1111111119389534 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4619206488132477 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.31862518191337585 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.21717485785484314 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4583333432674408 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.16212593019008636 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5208333730697632 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5441953539848328 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.008620689623057842 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9909822344779968 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.15603449940681458 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10592572391033173 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5530603528022766 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5564323663711548 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5320559740066528 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02006223239004612 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚jÁ&vù0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,jet/accuracy_4_of_4‚0.006574025843292475 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.026672380045056343 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.28333333134651184 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6041666269302368 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.25740739703178406 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4930993914604187 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.12231827527284622 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5921028852462769 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 6validation_average_jet_accuracy‚0.4570128917694092 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.535775899887085 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_3_of_7‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.22933152318000793 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 _runtime‚12903.160546296 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8924403190612793 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9903996586799622 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9979089498519897 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16590489447116852 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 *jet/accuracy_1_of_2‚0.7753903269767761 particle/accuracy_1_of_1‚1 jet/accuracy_4_of_7‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20790450274944305 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.029941076412796974 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5644018650054932 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5367523431777954 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_5_of_5‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.2395833432674408 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5934305191040039 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1111111119389534 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9894494414329529 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2983283996582031 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.47333332896232605 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0011337868636474013 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1993243247270584 particle/accuracy_1_of_4‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.562311589717865 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6233811974525452 8"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.444294273853302 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 )loss/val_total_loss‚40.59402007332412 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.38937899470329285 jet/accuracy_1_of_7‚0.5 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7085139751434326 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6041666269302368 +jet/accuracy_3_of_4‚0.07720115035772324 particle/accuracy_4_of_4‚1 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1506449580192566 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.38498759269714355 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.38564640283584595 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1145339161157608 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8991228342056274 jet/accuracy_5_of_7‚0 particle/accuracy_5_of_7‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5506362915039062 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4910675585269928 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5120073556900024 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12364259362220764 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.8571428656578064 * Purity/3FRt3SRqqt0FBt/FRt_purity‚0.125 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.33878448605537415 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.11897147446870804 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2884754538536072 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.6373456120491028 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5589009523391724 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.20354768633842468 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8"Purity/0FRt1SRqqt2FBt/event_purity‚0.071428582072258 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6044495701789856 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15463194251060486 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4435267746448517 *jet/accuracy_1_of_5‚0.9364311695098877 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.7251983880996704 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.4166666567325592 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9670635461807251 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.14487873017787933 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ( Purity/3FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6041666269302368 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.04463321343064308 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9918607473373413 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23404674232006073 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.4424917697906494 particle/accuracy_4_of_6‚1 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2069619745016098 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.18260066211223602 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5357142686843872 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 jet/accuracy_6_of_7‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2600456476211548 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23890389502048492 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 jet/accuracy_3_of_7‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,particle/sensitivity‚0.34980443120002747 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.43560609221458435 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3888888955116272 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2364199161529541 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.992834210395813 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9910677671432495 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12999886274337769 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 'particle/f_score‚0.4326569437980652 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_6‚0.9807692170143127 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.20011936128139496 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.012784897349774837 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4412015378475189 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5254629850387573 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.20011936128139496 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5298120379447937 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5519359707832336 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 particle/accuracy_1_of_7‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735632538795471 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15033183991909027 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.028212400153279305 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5843871831893921 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9322916865348816 , Purity/3FRt0SRqqt1FBt/FRt_purity‚0.46875 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.14900031685829163 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 particle/accuracy_3_of_3‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007607860025018454 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.11934155970811844 jet/accuracy_4_of_6‚0 jet/accuracy_2_of_7‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5921044945716858 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5765652060508728 ,jet/accuracy_4_of_5‚0.018164362758398056 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 particle/accuracy_1_of_2‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_6‚1 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.019607843831181526 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5458713173866272 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.37986111640930176 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.26802757382392883 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.20895594358444214 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_4‚0.9288938641548157 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.17555555701255798 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.46666663885116577 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9893516898155212 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.32590410113334656 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1710260659456253 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5878205299377441 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6542103886604309 particle/accuracy_2_of_4‚1 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 particle/accuracy_6_of_6‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5639155507087708 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0924656018614769 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17090784013271332 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4451970160007477 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5579234957695007 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.012152778916060925 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.5155172944068909 particle/accuracy_2_of_3‚1 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.019607843831181526 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1526380330324173 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1964285671710968 jet/accuracy_5_of_6‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3006852865219116 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9895833134651184 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16968944668769836 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5441953539848328 8"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.461952805519104 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9907757043838501 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6016176342964172 jet/accuracy_6_of_6‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4583333432674408 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 *jet/accuracy_2_of_5‚0.5477150082588196 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.23809263110160828 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5004075169563293 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03379314765334129 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20252405107021332 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9915520548820496 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3005827069282532 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19990846514701843 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 particle/accuracy_6_of_7‚1 % _timestamp‚1.7884768499732084e+09 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9845558404922485 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43209874629974365 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.18859650194644928 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 *jet/accuracy_2_of_6‚0.6474359035491943 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9184397459030151 )jet/accuracy_1_of_1‚0.509348452091217 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 jet/accuracy_7_of_7‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.04441244155168533 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 particle/accuracy_7_of_7‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0015873016091063619 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5207627415657043 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1226053535938263 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.07708333432674408 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 +"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.75 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.6681034564971924 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2615740895271301 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 (particle/accuracy‚0.8568275570869446 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 _step‚894 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5792391300201416 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5263062119483948 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_3_of_6‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.27003705501556396 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9979310631752014 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5164728164672852 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9981898069381714 particle/accuracy_4_of_5‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1490381956100464 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8219982385635376 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5192708373069763 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 epoch‚8 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.49714675545692444 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0025767881888896227 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21943941712379456 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.55418860912323 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9968157410621643 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5926398038864136 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.40903881192207336 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.42276665568351746 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 particle/accuracy_2_of_2‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5539477467536926 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1631205677986145 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.017643555998802185 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333432674408 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5734649300575256 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 particle/accuracy_5_of_6‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 6"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5213583111763 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.29542770981788635 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9938086867332458 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.14269006252288818 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 trainer/global_step‚10880 7"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.55501788854599 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9917800426483154 +jet/accuracy_3_of_5‚0.11566180735826492 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.170049250125885 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.5111110806465149 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7196969389915466 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.10721781849861145 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 +jet/accuracy_3_of_3‚0.03364181146025658 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.13636364042758942 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5852925181388855 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4014880955219269 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5458441376686096 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5468980669975281 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6437454223632812 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1944444477558136 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12875798344612122 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.05436229333281517 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5843355059623718 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4058696925640106 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9908068776130676 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.04848943278193474 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.36233067512512207 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5099999904632568 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9876037240028381 particle/accuracy_1_of_6‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4960766136646271 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9739024043083191 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4648818075656891 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.26522988080978394 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.14039744436740875 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚h±j¢0 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 * Purity/3FRt3SRqqt*FBt/FRt_purity‚0.125 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5575966835021973 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.054447438567876816 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_7‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.7017543911933899 +jet/accuracy_3_of_6‚0.21153846383094788 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9869740009307861 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5822916626930237 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0H:>šP{©,̦ >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4583333432674408 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 *jet/accuracy_2_of_5‚0.5477150082588196 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.23809263110160828 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5004075169563293 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03379314765334129 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20252405107021332 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9915520548820496 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3005827069282532 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19990846514701843 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 particle/accuracy_6_of_7‚1 % _timestamp‚1.7884768499732084e+09 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9845558404922485 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43209874629974365 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.18859650194644928 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 *jet/accuracy_2_of_6‚0.6474359035491943 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9184397459030151 )jet/accuracy_1_of_1‚0.509348452091217 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 jet/accuracy_7_of_7‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.04441244155168533 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 particle/accuracy_7_of_7‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0015873016091063619 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5207627415657043 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1226053535938263 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.07708333432674408 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 +"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.75 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.6681034564971924 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2615740895271301 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 (particle/accuracy‚0.8568275570869446 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 _step‚894 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5792391300201416 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5263062119483948 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_3_of_6‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.27003705501556396 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9979310631752014 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5164728164672852 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9981898069381714 particle/accuracy_4_of_5‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1490381956100464 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8219982385635376 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5192708373069763 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 epoch‚8 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.49714675545692444 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0025767881888896227 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21943941712379456 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.55418860912323 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9968157410621643 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5926398038864136 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.40903881192207336 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.42276665568351746 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 particle/accuracy_2_of_2‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5539477467536926 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1631205677986145 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.017643555998802185 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333432674408 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5734649300575256 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 particle/accuracy_5_of_6‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 6"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5213583111763 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.29542770981788635 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9938086867332458 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.14269006252288818 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 trainer/global_step‚10880 7"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.55501788854599 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9917800426483154 +jet/accuracy_3_of_5‚0.11566180735826492 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.170049250125885 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.5111110806465149 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7196969389915466 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.10721781849861145 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 +jet/accuracy_3_of_3‚0.03364181146025658 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.13636364042758942 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5852925181388855 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4014880955219269 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5458441376686096 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5468980669975281 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6437454223632812 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1944444477558136 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12875798344612122 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.05436229333281517 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5843355059623718 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4058696925640106 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9908068776130676 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.04848943278193474 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.36233067512512207 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5099999904632568 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9876037240028381 particle/accuracy_1_of_6‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4960766136646271 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9739024043083191 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4648818075656891 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.26522988080978394 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.14039744436740875 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 * Purity/3FRt3SRqqt*FBt/FRt_purity‚0.125 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5575966835021973 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.054447438567876816 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_7‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.7017543911933899 +jet/accuracy_3_of_6‚0.21153846383094788 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9869740009307861 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5822916626930237 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3888888955116272 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.49477121233940125 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.11269406974315643 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15659789741039276 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9897192716598511 8"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.581626296043396 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6386802196502686 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.35092586278915405 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5551514029502869 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0033104149624705315 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.17929291725158691 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.27369406819343567 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5693391561508179 particle/accuracy_2_of_5‚1 +particle/specificity‚0.9506559371948242 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1111111119389534 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4619206488132477 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.31862518191337585 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.21717485785484314 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4583333432674408 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.16212593019008636 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5208333730697632 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5441953539848328 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.008620689623057842 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9909822344779968 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.15603449940681458 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10592572391033173 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5530603528022766 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5564323663711548 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5320559740066528 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02006223239004612 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,jet/accuracy_4_of_4‚0.006574025843292475 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.026672380045056343 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.28333333134651184 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6041666269302368 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.25740739703178406 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4930993914604187 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.12231827527284622 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5921028852462769 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 6validation_average_jet_accuracy‚0.4570128917694092 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.535775899887085 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_3_of_7‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.22933152318000793 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 _runtime‚12903.160546296 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8924403190612793 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9903996586799622 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9979089498519897 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16590489447116852 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 *jet/accuracy_1_of_2‚0.7753903269767761 particle/accuracy_1_of_1‚1 jet/accuracy_4_of_7‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20790450274944305 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.029941076412796974 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5644018650054932 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5367523431777954 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_5_of_5‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.2395833432674408 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5934305191040039 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1111111119389534 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9894494414329529 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2983283996582031 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.47333332896232605 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0011337868636474013 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1993243247270584 particle/accuracy_1_of_4‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.562311589717865 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6233811974525452 8"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.444294273853302 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 )loss/val_total_loss‚40.59402007332412 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.38937899470329285 jet/accuracy_1_of_7‚0.5 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7085139751434326 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6041666269302368 +jet/accuracy_3_of_4‚0.07720115035772324 particle/accuracy_4_of_4‚1 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1506449580192566 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.38498759269714355 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.38564640283584595 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1145339161157608 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8991228342056274 jet/accuracy_5_of_7‚0 particle/accuracy_5_of_7‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5506362915039062 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4910675585269928 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5120073556900024 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12364259362220764 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.8571428656578064 * Purity/3FRt3SRqqt0FBt/FRt_purity‚0.125 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.33878448605537415 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.11897147446870804 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2884754538536072 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.6373456120491028 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5589009523391724 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.20354768633842468 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8"Purity/0FRt1SRqqt2FBt/event_purity‚0.071428582072258 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6044495701789856 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15463194251060486 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4435267746448517 *jet/accuracy_1_of_5‚0.9364311695098877 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.7251983880996704 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.4166666567325592 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9670635461807251 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.14487873017787933 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ( Purity/3FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6041666269302368 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.04463321343064308 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9918607473373413 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23404674232006073 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.4424917697906494 particle/accuracy_4_of_6‚1 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2069619745016098 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.18260066211223602 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5357142686843872 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 jet/accuracy_6_of_7‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2600456476211548 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23890389502048492 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 jet/accuracy_3_of_7‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,particle/sensitivity‚0.34980443120002747 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.43560609221458435 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3888888955116272 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2364199161529541 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.992834210395813 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9910677671432495 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12999886274337769 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 'particle/f_score‚0.4326569437980652 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_6‚0.9807692170143127 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.20011936128139496 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.012784897349774837 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4412015378475189 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5254629850387573 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.20011936128139496 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5298120379447937 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5519359707832336 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 particle/accuracy_1_of_7‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735632538795471 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15033183991909027 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.028212400153279305 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purityt‘-š/1FRt0SRqqt2FBt/FRt_purity‚0.5843871831893921 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9322916865348816 , Purity/3FRt0SRqqt1FBt/FRt_purity‚0.46875 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.14900031685829163 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 particle/accuracy_3_of_3‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007607860025018454 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.11934155970811844 jet/accuracy_4_of_6‚0 jet/accuracy_2_of_7‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5921044945716858 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5765652060508728 ,jet/accuracy_4_of_5‚0.018164362758398056 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 particle/accuracy_1_of_2‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_6‚1 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.019607843831181526 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5458713173866272 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.37986111640930176 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.26802757382392883 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.20895594358444214 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_4‚0.9288938641548157 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.17555555701255798 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.46666663885116577 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9893516898155212 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.32590410113334656 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1710260659456253 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5878205299377441 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6542103886604309 particle/accuracy_2_of_4‚1 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 particle/accuracy_6_of_6‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5639155507087708 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0924656018614769 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17090784013271332 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4451970160007477 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5579234957695007 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.012152778916060925 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.5155172944068909 particle/accuracy_2_of_3‚1 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.019607843831181526 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1526380330324173 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1964285671710968 jet/accuracy_5_of_6‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3006852865219116 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9895833134651184 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16968944668769836 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5441953539848328 8"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.461952805519104 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9907757043838501 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6016176342964172 jet/accuracy_6_of_6‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0þº–Û¼‚ª,‚(:÷ ½ûçÔ„‚¶ˆ disk.sda1.out‚0 disk.md0.out‚ 1375.453125 disk.rbd0.out‚ 3.646484375$ memory_percent‚40.37675857543945! disk.sda2.out‚1427.2958984375# disk.nvme1n1.in‚201440.03515625# disk.nvme0n1.out‚78090.41796875 cpu‚25.20928708279526# disk.nvme0n1.in‚330003.13671875 disk.md0.in‚ 10.98046875 network.sent‚ 2.9292383e+07 network.recv‚ 4.444853e+06 disk.sda2.in‚ 5.57421875 disk.md0p1.in‚ 10.98046875 disk.md0p1.out‚ 1375.453125 disk.rbd0.in‚0.015625( disk./.usagePercent‚83.1848668501114! disk.sdb2.out‚1427.2958984375 disk.rbd2.in‚0 disk.rbd2.out‚ 3.65869140625 disk.sdb.in‚5.40625 disk.sdb1.out‚0 disk.sda.out‚1427.2958984375) proc.memory.percent‚18.50731372833252$ disk./.usageGB‚2975.093765258789# disk.nvme1n1.out‚74052.10546875 disk.sdb1.in‚0 disk.dm-0.in‚ 531443.171875# proc.memory.rssMB‚ 3032.23828125 proc.cpu.threads‚17 disk.sdb.out‚1427.2958984375 disk.sda.in‚ 5.57421875 disk.sdb2.in‚5.40625 disk.sda1.in‚0 disk.dm-0.out‚152142.5234375' proc.memory.availableMB‚ 9768.671875šáÇ’«,‚(:‡ ½ûçÔÇìĈ gpu.0.gpu‚10.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚142.653' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.66325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0b–w4¬,/ _wandbruntime‚12928  _runtime‚12928=Qª<É­,à & lr-AdamW/pg1‚2.413280034357375e-05 & lr-AdamW/pg2‚2.413280034357375e-05 trainer/global_step‚10899 % _timestamp‚1.7884768754123156e+09 _runtime‚12928.597475296 _step‚895‰ XÝή,È & lr-AdamW/pg1‚2.413280034357375e-05 & lr-AdamW/pg2‚2.413280034357375e-05 trainer/global_step‚10899 % _timestamp‚1.7884768754123156e+09 _runtime‚12928.597475296 _step‚895ÿ‹p¤ò4¯,/ _wandbruntime‚12928  _runtime‚12928(Ésò'°,ìO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 296729621 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚466961809248256 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚466961557576704 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚687 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21679016932352 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚78966911 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚493632158618624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚10899 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 504661204 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 296729621 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚493632466886656 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21617547021520 % _timestamp‚1.7884768754137325e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚445282792315904 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 _step‚896 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 207931585 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 296755965 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21678783263744 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 124374212 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 207931585 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 296755965 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚466961557576704 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚445282792315904 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚445282774312960 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚466961809248256 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 203341810 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚685 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚78967596 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚445282774312960 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚423586823244144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚12928.599533055 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 207931583 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21678783263744 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 504687550 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 124374214 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22044955671040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 207931583 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚471587487275520 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 203341123 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21679016932352 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22044671343104 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21617318040436 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚423586806204784 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚471587511215616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚445204370265664 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚445204124245220 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 _step‚896 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 207931585 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 296755965 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21678783263744 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 124374212 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 207931585 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 296755965 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚466961557576704 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚445282792315904 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚445282774312960 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚466961809248256 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 203341810 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚685 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚78967596 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚445282774312960 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚423586823244144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚12928.599533055 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 207931583 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21678783263744 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 504687550 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 124374214 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22044955671040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 207931583 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚471587487275520 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 203341123 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21679016932352 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22044671343104 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21617318040436 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚423586806204784 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚471587511215616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚445204370265664 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚445204124245220 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552€øpè¼4²,/ _wandbruntime‚12929  _runtime‚12929GЀ³,öM PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚445322078352896 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 207949860 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 207949860 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 203358866 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884768765950503e+09 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚423624353601568 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚-(g|2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚445243859112282 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 504732543 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚78974503 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚897 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚445243613091630 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚467003059324416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚445322060349952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚493676172453888 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚10899 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21680747289600 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 504706171 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 296756313 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚493675853713408 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚467003059324416 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21680980971520 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21680980971520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 124385014 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 203359519 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21619276529422 _runtime‚12929.781967229 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21680747289600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚423624336562208 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 207949858 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 296756313 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚471629195185664 •sN'´,ûM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚653 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 124385016 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 504706171 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚651 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22046977268224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚445322060349952 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 207949858 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚78973852 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21619505510714 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 296782683 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 296782683 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚445322078352896 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 207949860 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 207949860 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 203358866 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884768765950503e+09 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚423624353601568 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚445243859112282 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 504732543 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚78974503 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚897 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚445243613091630 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚467003059324416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚445322060349952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚493676172453888 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚10899 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21680747289600 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 504706171 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 296756313 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚493675853713408 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚467003059324416 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21680980971520 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21680980971520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 124385014 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 203359519 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21619276529422 _runtime‚12929.781967229 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21680747289600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚423624336562208 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 207949858 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 296756313 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚471629195185664 PЛù½Ä,ëO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚447178580156928 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚651 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚473671709718016 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _runtime‚12988.148725481 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21777232638976 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 298093917 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚79315815 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22144805801472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21776998970368 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚469024075132928 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21776998970368 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22144523570688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚447247094165504 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚469024326804480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 298093917 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282230784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 298067573 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚469024326804480 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚447247076162560 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚447247094165504 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 298067573 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚469024075132928 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21715485062028 _step‚900 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 $ _timestamp‚1.788476934963675e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚649 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚425463341115344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 506939252 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚447178826177372 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚473671685777920 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚10949 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ?9DeviceSÔäc4 tatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 208845333 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚79316464 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306170880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084Û.à‡ö'Å,ðO UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 208845335 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 506912906 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚447247076162560 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 204230127 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 208845333 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚79316464 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306170880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚447178580156928 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚651 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚473671709718016 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _runtime‚12988.148725481 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21777232638976 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 298093917 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚79315815 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22144805801472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21776998970368 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚469024075132928 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21776998970368 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22144523570688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚447247094165504 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚469024326804480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 298093917 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282230784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 298067573 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚469024326804480 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚447247076162560 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚447247094165504 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 298067573 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚469024075132928 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21715485062028 _step‚900 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 $ _timestamp‚1.788476934963675e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚649 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚425463341115344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 506939252 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚447178826177372 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚473671685777920 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚10949 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 506984245 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚473713369747968 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚447286362199552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 124925116 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 506984245 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22146810064384 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚447218315023990 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21779196678144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚447218069003338 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 298120635 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚469065576880640 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚473713393688064 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚425500854433408 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21778962996224 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 208863610 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 204248315 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788476935987258e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21778962996224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 124925114 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 335517696 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 506957873 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 208863610 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 204247640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0iÎv„$È,úM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚447286380202496 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 204247640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚469065576880640 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚469065325195776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚79322526 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 506984245 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚473713369747968 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚447286362199552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 124925116 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 506984245 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22146810064384 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚447218315023990 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21779196678144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚447218069003338 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 298120635 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚469065576880640 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚473713393688064 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚425500854433408 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21778962996224 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 208863610 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 204248315 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788476935987258e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21778962996224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 124925114 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 335517696 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 506957873 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 208863610 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚449211396015104 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 125454412 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22244577642496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚427339858986544 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21875214676992 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚471086592689152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 299405525 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚498000485862912 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚449211378012160 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚427339841947184 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21813423102536 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚449211396015104 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 209759085 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚10999 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 209759083 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚449211378012160 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚449153282089080 _step‚904 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚471086844360704 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21875448345600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 205116409 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 125454414 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 299431869 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚498000175497728 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21875214676992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21875448345600 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 _runtime‚13047.758110539 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚449153036068636 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚79661311 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22244291217408 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚471086592689152 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21813194121452 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚686 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 % _timestamp‚1.7884769945730197e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 509190954 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309u¾÷'Ù,ñO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚475755908220416 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 509164608 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 209759083 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚79661995 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 299405525 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚471086844360704 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 205115723 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 209759085 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚449211396015104 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 125454412 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22244577642496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚427339858986544 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21875214676992 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚471086592689152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 299405525 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚498000485862912 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚449211378012160 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚427339841947184 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21813423102536 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚449211396015104 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 209759085 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚10999 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 209759083 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚449211378012160 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚449153282089080 _step‚904 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚471086844360704 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21875448345600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 205116409 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 125454414 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 299431869 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚498000175497728 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21875214676992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21875448345600 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 _runtime‚13047.758110539 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚449153036068636 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚79661311 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22244291217408 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚471086592689152 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21813194121452 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚686 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 % _timestamp‚1.7884769945730197e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 509190954 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚475755884280320ˆ)fF4Ú,/ _wandbruntime‚13048  _runtime‚13048Êo˜“Û,öM REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚79668088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 205133302 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 509235947 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 209777360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21815152610438 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 299432217 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚449192524915046 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 299458587 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚449250682052096 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 299432217 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 125465216 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚471128094436864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚449250664049152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚475797568250368 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 209777358 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚449192770935698 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚703 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚498043865218560 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚905 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚475797592190464 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21877412384768 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 209777360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 125465214 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚449250664049152 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21877178702848 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21877178702848 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21815381591730 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 509209575 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22246296968192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 509235947 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21877412384768 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 _runtime‚13048.700590685 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚471127842752000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚79668791 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22246593865728 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 209777358 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864«ø Œ'Ü,ûM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚475797592190464 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21877412384768 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 209777360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 125465214 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚449250664049152 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21877178702848 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21877178702848 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21815381591730 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 509209575 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22246296968192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 509235947 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21877412384768 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 _runtime‚13048.700590685 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚471127842752000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚79668791 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22246593865728 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 209777358 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚79668088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 205133302 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 509235947 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 209777360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21815152610438 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 299432217 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚449192524915046 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 299458587 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚449250682052096 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 299432217 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 125465216 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚471128094436864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚449250664049152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚475797568250368 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 209777358 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚449192770935698 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚703 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚498043865218560 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚905 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚451127738000788 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚473149110245376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 210672835 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚473149361916928 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21973664052224 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚451175679861760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 210672833 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚80005159 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 125994514 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚500184106798592 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 206000327 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚477840106722816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚656 _runtime‚13107.388648634 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 300743477 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚451127491980344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 511416310 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚473149361916928 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚477840082782720 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 24ü9eÅ06020444 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚429216376857744 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚654 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚500184421358080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚451175697864704 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 300769821 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚451175697864704 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 125994512 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _step‚908 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 300743477 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚473149110245376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7884770542031143e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 210672835 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22344024015872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21911361143044 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21973430383616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21911132161960 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚80005813 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0—Ж÷'í,ñO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 210672833 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚451127738000788 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚473149110245376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 210672835 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚473149361916928 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21973664052224 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚451175679861760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 210672833 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚80005159 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 125994514 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚500184106798592 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 206000327 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚477840106722816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚656 _runtime‚13107.388648634 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 300743477 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚451127491980344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 511416310 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚473149361916928 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚477840082782720 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚429216376857744 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚654 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚500184421358080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚451175697864704 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 300769821 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚451175697864704 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 125994512 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _step‚908 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 300743477 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚473149110245376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7884770542031143e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 210672835 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22344024015872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21911361143044 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21973430383616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21911132161960 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚80005813Œ*?D4î,/ _wandbruntime‚13108  _runtime‚13108CA•û&ï,õM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 300796539 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 300770169 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚80013254 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚473190360308224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22346010915840 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚80012574 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21913090650946 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚451214965898752 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21975628091392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 206018570 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 206017888 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚473190611993088 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 511487649 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 300796539 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚429253890175808 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 210691110 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 210691108 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21975394409472 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 511461277 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22346307813376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚13108.439234346 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 126005316 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚429253907215168 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21913319632238 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 126005314 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21975628091392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚500227777668608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚451166980826754 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚451214983901696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 511461277 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _step‚909 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21975394409472 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚680 $ _timestamp‚1.788477055253802e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚451214965898752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚473190611993088 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚500228098506240 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚477881790692864 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 trainer/global_step‚11049)AA‡ð,úM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 206017888 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚473190611993088 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 511487649 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 300796539 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚429253890175808 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 210691110 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 210691108 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21975394409472 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 511461277 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22346307813376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚13108.439234346 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 126005316 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚429253907215168 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21913319632238 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 126005314 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21975628091392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚500227777668608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚451166980826754 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚451214983901696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 511461277 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _step‚909 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21975394409472 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚680 $ _timestamp‚1.788477055253802e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚451214965898752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚473190611993088 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚500228098506240 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚477881790692864 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 trainer/global_step‚11049 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 300796539 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 300770169 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚80013254 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚473190360308224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22346010915840 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚80012574 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21913090650946 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocatpÿy ed_bytes.large_pool.freed‚451214965898752 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21975628091392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 206018570 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 71345111046m‚­4ñ,/ _wandbruntime‚13108  _runtime‚13108¿ù_ò,Ù 1loss/FBt2/assignment_loss‚0.16387645634961706 1loss/SRqqt1/detection_loss‚2.3273491280920364 /loss/FBt1/detection_loss‚2.3711833477020265 1loss/SRqqt2/assignment_loss‚1.900420561689755 2loss/FBt3/assignment_loss‚0.009722505936777518 1loss/FBt4/assignment_loss‚0.20602993938373046 .loss/FRt1/detection_loss‚2.840601136907935 .loss/FRt4/detection_loss‚4.177732944488525 %loss/total_loss‚41.59545740183226 epoch‚9 0loss/SRqqt4/detection_loss‚2.529568743892014 /loss/FBt2/detection_loss‚1.9580125948962044 trainer/global_step‚11049 _runtime‚13108.440822713 0loss/FRt1/assignment_loss‚2.9114280827343464 1loss/SRqqt4/assignment_loss‚1.449350130977109 % _timestamp‚1.7884770552561188e+09 _step‚910 .loss/FRt2/detection_loss‚2.150537287555612 1loss/SRqqt3/detection_loss‚0.9918136359916793 2loss/SRqqt3/assignment_loss‚0.9284100354105855 1loss/SRqqt2/detection_loss‚2.3269853838558854 0loss/FRt2/assignment_loss‚3.3981844829578027 /loss/FRt3/assignment_loss‚0.847024760463021 1loss/FBt1/assignment_loss‚0.10662890813538865 /loss/FRt3/detection_loss‚1.4504738910067263 0loss/FBt3/detection_loss‚0.49688770226799905 /loss/FBt4/detection_loss‚2.1282796382904055 /loss/FRt4/assignment_loss‚2.557578893808218 2loss/SRqqt1/assignment_loss‚1.3673772090388572ê¸|´dó,Þ .loss/FRt1/detection_loss‚2.840601136907935 .loss/FRt4/detection_loss‚4.177732944488525 %loss/total_loss‚41.59545740183226 epoch‚9 0loss/SRqqt4/detection_loss‚2.529568743892014 /loss/FBt2/detection_loss‚1.9580125948962044 trainer/global_step‚11049 _runtime‚13108.440822713 0loss/FRt1/assignment_loss‚2.9114280827343464 1loss/SRqqt4/assignment_loss‚1.449350130977109 % _timestamp‚1.7884770552561188e+09 _step‚910 .loss/FRt2/detection_loss‚2.150537287555612 1loss/SRqqt3/detection_loss‚0.9918136359916793 2loss/SRqqt3/assignment_loss‚0.9284100354105855 1loss/SRqqt2/detection_loss‚2.3269853838558854 0loss/FRt2/assignment_loss‚3.3981844829578027 /loss/FRt3/assignment_loss‚0.847024760463021 1loss/FBt1/assignment_loss‚0.10662890813538865 /loss/FRt3/detection_loss‚1.4504738910067263 0loss/FBt3/detection_loss‚0.49688770226799905 /loss/FBt4/detection_loss‚2.1282796382904055 /loss/FRt4/assignment_loss‚2.557578893808218 2loss/SRqqt1/assignment_loss‚1.3673772090388572 1loss/FBt2/assignment_loss‚0.16387645634961706 1loss/SRqqt1/detection_loss‚2.3273491280920364 /loss/FBt1/detection_loss‚2.3711833477020265 1loss/SRqqt2/assignment_loss‚1.900420561689755 2loss/FBt3/assignment_loss‚0.009722505936777518 1loss/FBt4/assignment_loss‚0.20602993938373046Ž£j+6’ô,‚(:‡ €ýçÔøÐû‹ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚130.841' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.71025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½]÷‰õ,‚(:þ €ýçÔÖãá‘" disk.sdb2.out‚1452.77783203125 disk.md0.in‚ 10.98046875 disk.md0.out‚ 1400.06640625# disk.nvme1n1.in‚201442.66796875 disk.sdb.in‚5.40625 disk.rbd0.out‚ 3.646484375 disk.sdb1.out‚0# disk.nvme0n1.out‚79117.24609375 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0p1.out‚ 1400.06640625 network.recv‚ 4.509805e+06) disk./.usagePercent‚83.17968090837911$ disk./.usageGB‚2974.908290863037 disk.rbd2.in‚0! disk.sdb.out‚1452.77783203125 disk.sda.in‚ 5.57421875 disk.sda2.in‚ 5.57421875 disk.md0p1.in‚ 10.98046875$ memory_percent‚41.42251014709473# proc.memory.rssMB‚ 3032.23828125 proc.cpu.threads‚17 disk.rbd2.out‚ 3.87109375! disk.sda.out‚1452.77783203125! disk.dm-0.out‚154712.06640625 network.sent‚ 2.9778186e+07 disk.rbd0.in‚0.015625 disk.sdb1.in‚0 disk.nvme0n1.in‚ 330006.21875( proc.memory.availableMB‚ 9597.3359375) proc.memory.percent‚18.50731372833252 cpu‚25.206639124380228 disk.sdb2.in‚5.40625" disk.nvme1n1.out‚ 75594.8203125 disk.dm-0.in‚531448.88671875" disk.sda2.out‚1452.77783203125¤•_ö,‚(:… ýçÔÚí‡ gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚90.846' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚22.7115 gpu.0.smClock‚1215 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿ÊGx÷,‚(:í ýçÔÈœ£ˆ disk.sda.in‚ 5.57421875) proc.memory.percent‚18.50731372833252 proc.cpu.threads‚17% disk./.usageGB‚2975.0962867736816! disk.nvme1n1.out‚ 75667.140625 disk.sdb.in‚5.40625 disk.sdb1.in‚0 disk.sda.out‚1454.5634765625 disk.sda1.in‚0 disk.md0p1.in‚ 10.98046875 network.sent‚ 2.9887401e+07 network.recv‚ 4.51624e+06 disk.dm-0.out‚ 154835.75 disk.md0.in‚ 10.98046875 disk.rbd2.in‚0# proc.memory.rssMB‚ 3032.23828125) disk./.usagePercent‚83.18493735272313 disk.dm-0.in‚531757.27734375 disk.sda1.out‚0$ memory_percent‚41.42262935638428 disk.sdb.out‚1454.5634765625! disk.sda2.out‚1454.5634765625 disk.sdb2.in‚5.40625 cpu‚25.206422611961248! disk.nvme1n1.in‚ 201569.078125! disk.sdb2.out‚1454.5634765625 disk.sdb1.out‚0# disk.nvme0n1.in‚330188.19921875 disk.md0.out‚ 1401.78125 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 3.646484375 disk.rbd2.out‚ 3.87109375! disk.nvme0n1.out‚ 79168.609375 disk.sda2.in‚ 5.57421875 disk.md0p1.out‚ 1401.78125) proc.memory.availableMB‚ 9597.31640625ÿÙ³j’ø,‚(:‡ žýçÔù‡£‡ gpu.0.gpu‚10.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚143.595' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.89875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ(w†ù,‚(:ö žýçÔŽ§·ˆ$ memory_percent‚41.42260551452637 cpu‚25.20622512910424 network.sent‚ 2.9891518e+07$ disk./.usageGB‚2975.095733642578 disk.sda1.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda.in‚ 5.57421875! disk.nvme1n1.in‚ 201569.078125 network.recv‚ 4.520969e+06) disk./.usagePercent‚83.18492188694599# disk.nvme0n1.in‚330188.19921875 disk.dm-0.in‚531757.27734375 disk.md0p1.in‚ 10.98046875! disk.sdb2.out‚1457.1259765625 disk.md0.out‚ 1404.265625# proc.memory.rssMB‚ 3032.23828125 disk.md0p1.out‚ 1404.265625 disk.md0.in‚ 10.98046875 disk.rbd2.out‚ 3.87109375" disk.nvme1n1.out‚ 75811.2265625 proc.cpu.threads‚17! disk.sda2.out‚1457.1259765625 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.sdb.out‚1457.1259765625 disk.sda.out‚1457.1259765625# disk.nvme0n1.out‚79244.91015625 disk.sdb2.in‚5.40625) proc.memory.percent‚18.50731372833252 disk.sda1.out‚0 disk.sdb.in‚5.40625( proc.memory.availableMB‚ 9597.3203125! disk.dm-0.out‚155056.13671875 disk.sda2.in‚ 5.57421875 disk.rbd0.out‚ 3.646484375H‘ú,‚(:† ­ýçÔ˜ãÞ‡ gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚145.098' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.2745 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÍV´€û,‚(:õ ­ýçԧݳˆ disk.rbd0.in‚0.015625 disk.rbd2.out‚ 3.87109375 cpu‚25.206028314724435 disk.sdb2.out‚1458.673828125 disk.md0.out‚ 1405.7421875 disk.sda2.out‚1458.673828125 disk.md0p1.in‚ 10.98046875 disk.md0p1.out‚ 1405.7421875) proc.memory.percent‚18.50731372833252! disk.nvme1n1.in‚ 201569.078125 network.sent‚ 2.9899557e+07 network.recv‚ 4.525225e+06 disk.rbd0.out‚ 3.646484375 disk.sda.out‚1458.673828125 disk.dm-0.in‚531757.27734375! disk.dm-0.out‚155257.08203125 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sdb.out‚1458.673828125$ memory_percent‚41.42255783081055# proc.memory.rssMB‚ 3032.23828125 disk.sda1.out‚0) disk./.usagePercent‚83.15525095345983 disk.sdb1.in‚0 disk.sda.in‚ 5.57421875# disk.nvme0n1.in‚330188.19921875 disk.sdb1.out‚0 disk.sda2.in‚ 5.57421875 disk.sdb2.in‚5.40625 disk.sdb.in‚5.40625' proc.memory.availableMB‚ 9597.328125 disk.md0.in‚ 10.98046875" disk.nvme0n1.out‚ 79372.8359375 disk.sda1.in‚0# disk.nvme1n1.out‚75884.24609375% disk./.usageGB‚2974.0345573425293¥Ã¯4ü,/ _wandbruntime‚13162  _runtime‚13162‚ÎRøËý,Å ' lr-AdamW/pg1‚1.6759084660759556e-05 ' lr-AdamW/pg2‚1.6759084660759556e-05 trainer/global_step‚11099 % _timestamp‚1.7884771095958848e+09 _runtime‚13162.780820063 _step‚911 O uÐþ,Ê _step‚911 ' lr-AdamW/pg1‚1.6759084660759556e-05 ' lr-AdamW/pg2‚1.6759084660759556e-05 trainer/global_step‚11099 % _timestamp‚1.7884771095958848e+09 _runtime‚13162.780820063QÑž4ÿ,/ _wandbruntime‚13162  _runtime‚131621·K¢ò'€-ìO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚80353616 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚453139999714304 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 206887550 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 211586585 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚453101947892052 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22009299183552 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚678 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚502368585155072 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22444279929856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22071879758848 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚80352938 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 _step‚912 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 513668012 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22009070202468 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚475211627801600 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22071879758848 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 302081429 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚479924281285120 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 302107773 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 126534614 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 % _timestamp‚1.7884771095968997e+09 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 513694358 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚479924305225216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 trainer/global_step‚11099 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 302107773 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 211586585 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22071646090240 _runtime‚13162.782013999 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚475211879473152 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚475211627801600 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 126534612 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚453139981711360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608àˆ²iÖ'-ñO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22071646090240 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚475211627801600 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22071879758848 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 302081429 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚479924281285120 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 302107773 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 126534614 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 % _timestamp‚1.7884771095968997e+09 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 513694358 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚479924305225216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 trainer/global_step‚11099 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 302107773 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 211586585 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22071646090240 _runtime‚13162.782013999 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚475211879473152 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚475211627801600 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 126534612 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚453139981711360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚80353616 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚453139999714304 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 206887550 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 211586585 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚453101947892052 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22009299183552 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚678 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚502368585155072 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22444279929856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22071879758848 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚80352938 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 _step‚912 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 513668012 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22009070202468 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 trainer/global_step‚11099 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 211604860 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚453179285751296 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚475252877864448 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 302134491 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚479965989195264 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 513739351 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22073843798016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 513712979 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22073610116096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚431130408047008 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22011257672746 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚688 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚690 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 206905571 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 513739351 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 302108121 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚913 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚80360157 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚453141682759114 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22073610116096 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 206906261 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚431130425086368 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚479965965255168 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22445981151232 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 302108121 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 126545416 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 211604860 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 211604858 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚453179285751296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚80360845 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚453141436738462 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚502412258855424 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚502411946406400 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22011028691454 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚475253129549312 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚475253129549312 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 302134491 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22073843798016 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 211604858 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22446269660160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 trainer/global_step‚11099 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 211604860 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚453179285751296 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚475252877864448 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 302134491 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚479965989195264 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 513739351 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22073843798016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 513712979 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22073610116096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚431130408047008 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22011257672746 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚688 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚690 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 206905571 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 513739351 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 302108121 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚913 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚80360157 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚453141682759114 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22073610116096 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 206906261 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚431130425086368 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚479965965255168 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22445981151232 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 302108121 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 126545416 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 211604860 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 211604858 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚453179285751296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚80360845 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚453141436738462 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚502412258855424 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚502411946406400 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22011028691454‘óó34…-/ _wandbruntime‚13163  _runtime‚13163„Šê²\†-Ö 0loss/SRqqt1/detection_loss‚1.819639663363612 0loss/FBt3/detection_loss‚0.45552876163415157 trainer/global_step‚11099 .loss/FRt1/detection_loss‚2.578213868997036 /loss/FRt3/detection_loss‚1.4743579846690409 -loss/FBt1/detection_loss‚3.08691119402647 % _timestamp‚1.7884771105304413e+09 1loss/SRqqt1/assignment_loss‚1.336188803838436 2loss/SRqqt3/assignment_loss‚0.9294775514115047 .loss/FRt4/detection_loss‚2.732810892377581 2loss/FBt1/assignment_loss‚0.009360436888648849 1loss/FBt3/assignment_loss‚0.08063953863110158 epoch‚9 /loss/FRt1/assignment_loss‚2.921090925351167 0loss/FRt3/assignment_loss‚1.0377354472875595 /loss/FRt4/assignment_loss‚2.764595069204058 1loss/SRqqt4/detection_loss‚2.4558484392861524 _runtime‚13163.715151076 _step‚914 1loss/SRqqt3/detection_loss‚0.7984465158221369 2loss/SRqqt2/assignment_loss‚2.0553449910993757 1loss/SRqqt2/detection_loss‚2.3156484204989214 /loss/FBt2/detection_loss‚1.9141612138066972 -loss/FBt4/detection_loss‚2.25915664434433 %loss/total_loss‚40.07010767371884 0loss/FRt2/assignment_loss‚3.4193387554616343 2loss/SRqqt4/assignment_loss‚1.3513064393773675 1loss/FBt2/assignment_loss‚0.07065192523318028 1loss/FBt4/assignment_loss‚0.09674848452310211 .loss/FRt2/detection_loss‚2.106905706585575‰E?a‡-Û 2loss/FBt1/assignment_loss‚0.009360436888648849 1loss/FBt3/assignment_loss‚0.08063953863110158 epoch‚9 /loss/FRt1/assignment_loss‚2.921090925351167 0loss/FRt3/assignment_loss‚1.0377354472875595 /loss/FRt4/assignment_loss‚2.764595069204058 1loss/SRqqt4/detection_loss‚2.4558484392861524 _runtime‚13163.715151076 _step‚914 1loss/SRqqt3/detection_loss‚0.7984465158221369 2loss/SRqqt2/assignment_loss‚2.0553449910993757 1loss/SRqqt2/detection_loss‚2.3156484204989214 /loss/FBt2/detection_loss‚1.9141612138066972 -loss/FBt4/detection_loss‚2.25915664434433 %loss/total_loss‚40.07010767371884 0loss/FRt2/assignment_loss‚3.4193387554616343 2loss/SRqqt4/assignment_loss‚1.3513064393773675 1loss/FBt2/assignment_loss‚0.07065192523318028 1loss/FBt4/assignment_loss‚0.09674848452310211 .loss/FRt2/detection_loss‚2.106905706585575 0loss/SRqqt1/detection_loss‚1.819639663363612 0loss/FBt3/detection_loss‚0.45552876163415157 trainer/global_step‚11099 .loss/FRt1/detection_loss‚2.578213868997036 /loss/FRt3/detection_loss‚1.4743579846690409 -loss/FBt1/detection_loss‚3.08691119402647 % _timestamp‚1.7884771105304413e+09 1loss/SRqqt1/assignment_loss‚1.336188803838436 2loss/SRqqt3/assignment_loss‚0.9294775514115047 .loss/FRt4/detection_loss‚2.732810892377581’S}kb‘ˆ-‚(:† ¼ýçÔÐ÷¾‡ gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚107.002' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚26.7505 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0KÞNû‚‰-‚(:÷ ¼ýçÔ¿ÖÒˆ network.sent‚ 2.9916271e+07 disk.sdb1.out‚0" disk.nvme0n1.out‚ 79576.1640625 disk.sda2.in‚ 5.57421875" disk.sdb2.out‚1460.47607421875 disk.md0.in‚ 10.98046875 disk.rbd2.in‚0 cpu‚25.205812203397606 network.recv‚ 4.52963e+06 disk.sdb.in‚5.40625! disk.sda.out‚1460.47607421875 disk.sda1.in‚0$ memory_percent‚41.42134189605713 disk.rbd2.out‚ 3.87109375! disk.sdb.out‚1460.47607421875# disk.nvme0n1.in‚330188.19921875 disk.md0p1.in‚ 10.98046875 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 3.646484375# proc.memory.rssMB‚ 3032.23828125 disk.dm-0.in‚531757.27734375 disk.dm-0.out‚ 155581.75 disk.sdb2.in‚5.40625) proc.memory.availableMB‚ 9597.52734375) proc.memory.percent‚18.50731372833252) disk./.usagePercent‚83.15336572855458% disk./.usageGB‚2973.9671325683594! disk.nvme1n1.in‚ 201569.078125" disk.nvme1n1.out‚ 76005.5859375 disk.sda.in‚ 5.57421875 disk.md0p1.out‚ 1407.4921875 disk.md0.out‚ 1407.4921875 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.sda1.out‚0" disk.sda2.out‚1460.47607421875ìæ)TŽŠ-‚(:ƒ ËýçÔ‚Ú¡† gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚137.36' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.34 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ݶê샋-‚(:ø ËýçÔò´æ‡! disk.nvme1n1.in‚ 201569.078125 disk.sdb1.in‚0 disk.dm-0.out‚155624.9296875 disk.sdb2.in‚5.40625" disk.sdb2.out‚1462.20751953125 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 3.646484375 disk.sda.in‚ 5.57421875# disk.nvme0n1.in‚330188.19921875 disk.dm-0.in‚531757.27734375 disk.sda1.in‚0 disk.md0p1.in‚ 10.98046875 disk.md0p1.out‚ 1409.171875 disk.sdb.in‚5.40625 cpu‚25.20561988294933& disk./.usagePercent‚83.14754792983$ disk./.usageGB‚2973.759059906006# disk.nvme0n1.out‚79619.18359375 disk.rbd2.in‚0# proc.memory.rssMB‚ 3032.23828125) proc.memory.percent‚18.50731372833252! disk.sdb.out‚1462.20751953125 network.sent‚ 3.0012613e+07 network.recv‚ 4.535665e+06! disk.sda.out‚1462.20751953125 disk.sda2.in‚ 5.57421875 disk.md0.out‚ 1409.171875 disk.rbd2.out‚ 3.87109375 disk.sdb1.out‚0 disk.sda1.out‚0" disk.sda2.out‚1462.20751953125 disk.md0.in‚ 10.98046875# disk.nvme1n1.out‚76005.74609375$ memory_percent‚41.42439365386963) proc.memory.availableMB‚ 9597.02734375 proc.cpu.threads‚17K6ÒC‘Œ-‚(:† ÚýçÔý¾Ü‡ gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚133.958' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.4895 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k¹-RŒ-‚(:  ÚýçÔðｈ! disk.sdb.out‚1464.64111328125! disk.sda.out‚1464.64111328125 disk.sda1.in‚0 network.recv‚ 4.540238e+06 disk.sdb.in‚5.40625# disk.nvme0n1.out‚79650.14453125 disk.dm-0.out‚155625.6953125 disk.rbd0.in‚0.015625) proc.memory.percent‚18.50731372833252) disk./.usagePercent‚83.14819151948082 disk.sdb1.in‚0# disk.nvme0n1.in‚330190.34765625 disk.sda2.in‚ 5.57421875 disk.sdb2.in‚5.40625 disk.rbd0.out‚ 3.646484375$ memory_percent‚41.42289161682129 cpu‚25.205384091763825" disk.sda2.out‚1464.64111328125" disk.sdb2.out‚1464.64111328125 disk.md0.in‚ 10.98046875 disk.md0.out‚ 1411.54296875( proc.memory.availableMB‚ 9597.2734375 disk.md0p1.in‚ 10.98046875 disk.rbd2.out‚ 3.87109375# disk.nvme1n1.in‚201577.55078125 disk.rbd2.in‚0 network.sent‚ 3.0019017e+07 disk.sdb1.out‚0 disk.dm-0.in‚531767.8984375% disk./.usageGB‚2973.7820777893066 disk.sda.in‚ 5.57421875 disk.sda1.out‚0 disk.md0p1.out‚ 1411.54296875# disk.nvme1n1.out‚76005.92578125# proc.memory.rssMB‚ 3032.23828125 proc.cpu.threads‚17£>õm’Ž-‚(:‡ éýçÔŒÃÒ‡ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚124.321' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.08025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CFËw-‚(:ì éýçÔº¿‰ˆ disk.md0p1.in‚ 10.98046875 disk.md0p1.out‚ 1413.04296875 disk.sdb2.out‚1466.201171875 disk.rbd0.in‚0.015625 disk.sda.in‚ 5.57421875 disk.dm-0.in‚531776.3359375) proc.memory.percent‚18.50731372833252 proc.cpu.threads‚17 disk.rbd0.out‚ 3.646484375 disk.sdb.out‚1466.201171875 disk.sdb1.in‚0 disk.dm-0.out‚155698.5078125 disk.sda2.out‚1466.201171875 disk.rbd2.out‚ 3.87109375 disk.sdb1.out‚0) proc.memory.availableMB‚ 9597.52734375# proc.memory.rssMB‚ 3032.23828125) disk./.usagePercent‚83.14475086403564 disk.sdb2.in‚5.40625 disk.md0.in‚ 10.98046875 disk.nvme0n1.in‚330194.5 disk.rbd2.in‚0 disk.sda1.in‚0 network.sent‚ 3.0024894e+07 network.recv‚ 4.544406e+06 disk.sdb.in‚5.40625" disk.nvme0n1.out‚ 79690.8203125 disk.sda2.in‚ 5.57421875 disk.md0.out‚ 1413.04296875" disk.nvme1n1.in‚201581.8359375 disk.nvme1n1.out‚ 76007.6875 disk.sda.out‚1466.201171875 disk.sda1.out‚0$ disk./.usageGB‚2973.659023284912$ memory_percent‚41.42134189605713 cpu‚25.205190905066583pRÑ4-/ _wandbruntime‚13216  _runtime‚13216DU†Ë‘-Å _step‚915 ' lr-AdamW/pg1‚1.5120774857031939e-05 ' lr-AdamW/pg2‚1.5120774857031939e-05 trainer/global_step‚11149 % _timestamp‚1.7884771628983395e+09 _runtime‚13216.083393644±?n®Ð’-Ê % _timestamp‚1.7884771628983395e+09 _runtime‚13216.083393644 _step‚915 ' lr-AdamW/pg1‚1.5120774857031939e-05 ' lr-AdamW/pg2‚1.5120774857031939e-05 trainer/global_step‚11149“èb`T4“-/ _wandbruntime‚13216  _runtime‚13216ºu`6è ”-ìO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚80702855 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚455104301563904 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚455104283560960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 515946060 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 207778239 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚80703525 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚482008503727616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚916 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 212500335 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚672 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚432969395560784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 _runtime‚13216.084100777 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_byte3£ùE s.small_pool.current‚ 228981084 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 212500335 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 212500333 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 303445725 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22170095465472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 303445725 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 127074714 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚477274397029376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚455104283560960 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚504552377869312 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7884771628990633e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚477274397029376 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 303419381 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚477274145357824 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚477274145357824 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚482008479787520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚455076649824204 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22544184506880 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚455076403803760 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22170095465472 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 207777567 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 303419381 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496DÏØ÷'•-ñO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚477274145357824 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚477274145357824 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚482008479787520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚455076649824204 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22544184506880 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚455076403803760 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22170095465472 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 207777567 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 303419381 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚80702855 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚455104301563904 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚455104283560960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 515946060 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 207778239 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚80703525 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚482008503727616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚916 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 212500335 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚672 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚432969395560784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 _runtime‚13216.084100777 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 212500335 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 212500333 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 303445725 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22170095465472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 303445725 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 127074714 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚477274397029376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚455104283560960 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚504552377869312 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7884771628990633e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚477274397029376 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 303419381 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 207796074 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚504596371372032 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 212518610 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚477315395420672 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 515991053 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22172059504640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 trainer/global_step‚11149 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 207795387 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22108966731962 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚482050187697664 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 212518608 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 _runtime‚13217.020192285 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22172059504640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 515991053 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚685 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22171825822720 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22546183674368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 515964681 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚455143569597952 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 303472443 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚477315647105536 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 303472443 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚80710558 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚687 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 212518610 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 212518608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 515964681 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 127085516 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _step‚917 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 303446073 $ _timestamp‚1.788477163834533e+09 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 303446073 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚433006925918208 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22171825822720 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 127085514 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚504596046340096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0Z*ÿs­˜-úM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚477315395420672 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 515991053 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22172059504640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 trainer/global_step‚11149 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 207795387 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22108966731962 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚482050187697664 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 212518608 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 _runtime‚13217.020192285 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22172059504640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 515991053 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚685 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22171825822720 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22546183674368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 515964681 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚455143569597952 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 303472443 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚477315647105536 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 303472443 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚80710558 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚687 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 212518610 OADeviceStatsMonitor.on_train_ba¥,Stch_end/allocation.large_pool.freed‚ 212518608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 515964681 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 127085516 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _step‚917 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 303446073 $ _timestamp‚1.788477163834533e+09 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 303446073 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚433006925918208 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22171825822720 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 127085514 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚504596046340096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚455143587600896 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚455143569597952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22545882582528 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 301091840 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚455116138670822 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22109195713254 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚455143587600896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 207796074 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚504596371372032 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 212518610•Z³ó4™-/ _wandbruntime‚13217  _runtime‚13217^š-Ø 1loss/FBt2/assignment_loss‚0.07172904240912552 1loss/FBt4/assignment_loss‚0.13136360272762887 .loss/FBt4/detection_loss‚2.483954828977585 /loss/FRt1/assignment_loss‚2.650013470649719 0loss/FRt3/assignment_loss‚1.3467878646084241 2loss/SRqqt3/assignment_loss‚1.1485007864269476 .loss/FRt4/detection_loss‚2.988898859870049 2loss/SRqqt1/assignment_loss‚1.3867123845136828 .loss/FRt2/detection_loss‚2.053617435216438 /loss/FBt1/detection_loss‚3.3318343503134593 .loss/FBt2/detection_loss‚2.234930904015251 % _timestamp‚1.7884771638364294e+09 0loss/FBt1/assignment_loss‚0.2227646563467179 .loss/FRt1/detection_loss‚3.104832883675893 /loss/FRt3/detection_loss‚1.7200310851912946 1loss/SRqqt2/detection_loss‚2.0062474727630617 2loss/SRqqt4/assignment_loss‚1.0921156657369513 _runtime‚13217.02116114 trainer/global_step‚11149 1loss/FBt3/assignment_loss‚0.07192855536386768 1loss/SRqqt3/detection_loss‚0.7262268189302424 0loss/SRqqt4/detection_loss‚3.343575783466038 &loss/total_loss‚42.690135068098684 0loss/FRt2/assignment_loss‚3.3169157453205274 0loss/FRt4/assignment_loss‚3.1105762527834986 2loss/SRqqt2/assignment_loss‚1.5736620561530192 1loss/SRqqt1/detection_loss‚2.1659524357981152 epoch‚9 0loss/FBt3/detection_loss‚0.40696212684114774 _step‚918`¸’c›-Ý 0loss/FBt3/detection_loss‚0.40696212684114774 _step‚918 1loss/FBt2/assignment_loss‚0.07172904240912552 1loss/FBt4/assignment_loss‚0.13136360272762887 .loss/FBt4/detection_loss‚2.483954828977585 /loss/FRt1/assignment_loss‚2.650013470649719 0loss/FRt3/assignment_loss‚1.3467878646084241 2loss/SRqqt3/assignment_loss‚1.1485007864269476 .loss/FRt4/detection_loss‚2.988898859870049 2loss/SRqqt1/assignment_loss‚1.3867123845136828 .loss/FRt2/detection_loss‚2.053617435216438 /loss/FBt1/detection_loss‚3.3318343503134593 .loss/FBt2/detection_loss‚2.234930904015251 % _timestamp‚1.7884771638364294e+09 0loss/FBt1/assignment_loss‚0.2227646563467179 .loss/FRt1/detection_loss‚3.104832883675893 /loss/FRt3/detection_loss‚1.7200310851912946 1loss/SRqqt2/detection_loss‚2.0062474727630617 2loss/SRqqt4/assignment_loss‚1.0921156657369513 _runtime‚13217.02116114 trainer/global_step‚11149 1loss/FBt3/assignment_loss‚0.07192855536386768 1loss/SRqqt3/detection_loss‚0.7262268189302424 0loss/SRqqt4/detection_loss‚3.343575783466038 &loss/total_loss‚42.690135068098684 0loss/FRt2/assignment_loss‚3.3169157453205274 0loss/FRt4/assignment_loss‚3.1105762527834986 2loss/SRqqt2/assignment_loss‚1.5736620561530192 1loss/SRqqt1/detection_loss‚2.1659524357981152 epoch‚9–Š:ü¯’œ-‚(:‡ øýçÔÍÔÀˆ gpu.0.gpu‚11.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚145.771' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.44275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™Áºt-‚(:é øýçÔ÷Ãôˆ$ disk./.usageGB‚2973.783706665039 disk.md0.in‚ 10.98046875' proc.memory.availableMB‚ 9596.921875# proc.memory.rssMB‚ 3032.23828125! disk.dm-0.out‚155800.47265625 disk.rbd2.out‚ 3.87109375) proc.memory.percent‚18.50731372833252 disk.rbd0.out‚ 3.646484375 disk.rbd2.in‚0 network.sent‚ 3.0041531e+07 disk.sdb2.out‚1468.037109375 network.recv‚ 4.54846e+06) disk./.usagePercent‚83.14823706352804 disk.sdb1.in‚0 disk.sda.in‚ 5.57421875 disk.md0.out‚ 1414.8203125 cpu‚25.204974398848535 proc.cpu.threads‚17 disk.sda1.out‚0 disk.md0p1.in‚ 10.98046875 disk.md0p1.out‚ 1414.8203125# disk.nvme0n1.out‚79792.60546875 disk.sdb2.in‚5.40625 disk.sdb.in‚5.40625 disk.sdb.out‚1468.037109375 disk.sdb1.out‚0 disk.sda.out‚1468.037109375 disk.nvme0n1.in‚330194.5 disk.sda1.in‚0 disk.sda2.in‚ 5.57421875" disk.nvme1n1.out‚ 76007.8671875 disk.dm-0.in‚ 531780.234375 disk.sda2.out‚1468.037109375 disk.rbd0.in‚0.015625! disk.nvme1n1.in‚ 201585.734375# memory_percent‚41.4250373840332Tê%V‘ž-‚(:† ‡þçÔžÓ͇ gpu.0.gpu‚38.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚97.365' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚24.34125 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hVÅŒuŸ-‚(:ê ‡þçÔ„™Šˆ disk.md0p1.out‚ 1417.0078125 disk.md0.out‚ 1417.0078125 disk.sdb1.out‚0 disk.sda2.in‚ 5.57421875 disk.md0p1.in‚ 10.98046875 disk.rbd2.in‚0! disk.sda.out‚1470.29052734375 disk.nvme0n1.in‚330194.5 disk.dm-0.in‚ 531780.234375 disk.sdb2.in‚5.40625$ memory_percent‚41.42656326293945# proc.memory.rssMB‚ 3032.23828125) proc.memory.percent‚18.50731372833252! disk.nvme1n1.in‚ 201585.734375 disk.sdb1.in‚0 disk.sda.in‚ 5.57421875! disk.nvme0n1.out‚ 80012.859375 disk.sda1.in‚0" disk.sdb2.out‚1470.29052734375 cpu‚25.204764957419595) disk./.usagePercent‚83.15680158427583$ disk./.usageGB‚2974.090015411377 network.recv‚ 4.554955e+06 disk.rbd0.out‚ 3.646484375 disk.rbd2.out‚ 3.87109375 disk.sdb.in‚5.40625 disk.md0.in‚ 10.98046875' proc.memory.availableMB‚ 9596.671875" disk.sda2.out‚1470.29052734375 network.sent‚ 3.0137902e+07 disk.rbd0.in‚0.015625 disk.nvme1n1.out‚76008.25! disk.sdb.out‚1470.29052734375 disk.dm-0.out‚ 156021.109375 disk.sda1.out‚0 proc.cpu.threads‚17Tž -‚(:… –þçÔÖ‡ gpu.0.gpu‚24.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚143.872' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.968 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uáà{‡¡-‚(:ü –þçÔ°‡ˆ disk.rbd0.in‚0.015625# disk.nvme1n1.out‚76008.36328125! disk.dm-0.out‚156162.37109375 disk.md0p1.out‚ 1418.6953125 proc.cpu.threads‚17) disk./.usagePercent‚83.15679763783612 disk.sdb.in‚ 5.43359375 disk.sda2.in‚ 5.66796875) proc.memory.availableMB‚ 9596.67578125 cpu‚25.204571369176254" disk.nvme0n1.out‚ 80154.0078125 disk.sda1.in‚0! disk.sda2.out‚1472.0322265625 network.sent‚ 3.0144188e+07 disk.rbd0.out‚ 3.646484375 disk.sdb1.in‚0" disk.nvme0n1.in‚330194.6484375$ disk./.usageGB‚2974.089874267578 disk.md0.in‚ 11.1015625 disk.md0.out‚ 1418.6953125 disk.sda1.out‚0 disk.sda.out‚1472.0322265625 disk.md0p1.in‚ 11.1015625# proc.memory.rssMB‚ 3032.23828125 disk.sdb2.in‚ 5.43359375 disk.rbd2.in‚0! disk.nvme1n1.in‚ 201585.734375 network.recv‚ 4.559285e+06 disk.rbd2.out‚ 3.87109375 disk.sdb1.out‚0 disk.sda.in‚ 5.66796875$ memory_percent‚41.42653942108154! disk.sdb2.out‚1472.0322265625 disk.sdb.out‚1472.0322265625 disk.dm-0.in‚531780.3828125) proc.memory.percent‚18.50731372833252N©pV4¢-/ _wandbruntime‚13269  _runtime‚13269—€ÂÈ£- & lr-AdamW/pg1‚1.356522126019155e-05 & lr-AdamW/pg2‚1.356522126019155e-05 trainer/global_step‚11199 $ _timestamp‚1.788477216086039e+09 _runtime‚13269.271789302 _step‚919éULBͤ-Ç & lr-AdamW/pg1‚1.356522126019155e-05 & lr-AdamW/pg2‚1.356522126019155e-05 trainer/global_step‚11199 $ _timestamp‚1.788477216086039e+09 _runtime‚13269.271789302 _step‚919—ty[M4¥-/ _wandbruntime‚13269  _runtime‚13269„(Í÷ò'¦-ìO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22643690745344 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22268311172096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 304757333 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 213414085 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚479336662914048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚457068603413504 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884772160868504e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22268077503488 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22643983461888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 518197762 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚506736369035264 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _runtime‚13269.272187874 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚457068585410560 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22205175264568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚11199 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚484092702230016 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚479336662914048 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22268077503488 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚434845913431984 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22268311172096 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22204946283484 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 213414083 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 304783677 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚457068585410560 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 304757333 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚434845930471344 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 127614814 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚81047667 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 208662479 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚484092678289920 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚457068603413504 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 127614812 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚690 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 213414083 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚506736685691904 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚692 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚06¨¡Ò6&§-ñO \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚434845913431984 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22268311172096 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22204946283484 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 213414083 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 304783677 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚457068585410560 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 304757333 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚434845930471344 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 127614814 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚81047667 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 208662479 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚484092678289920 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚457068603413504 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 127614812 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚690 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 213414083 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚506736685691904 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚692 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22643690745344 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22268311172096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 304757333 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 213414085 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚479336662914048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚457068603413504 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884772160868504e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22268077503488 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22643983461888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 518197762 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚506736369035264 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _runtime‚13269.272187874 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚457068585410560 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22205175264568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚11199 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚484092702230016 GADeviceStatsMonitor.on_train_batch_start/over…¼GÁsize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚479336662914048 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22268077503488˜9¬ù34¨-/ _wandbruntime‚13270  _runtime‚13270Á]Íü&©-öM L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚457107889450496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚713 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 518216383 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 208679869 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚711 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚484134386200064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22645688009216 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚479378164661760 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 _runtime‚13270.213954404 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 213432360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚484134362259968 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22270275211264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚479378164661760 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 213432358 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 213432360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 % _timestamp‚1.7884772170283399e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 304810395 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 127625616 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 trainer/global_step‚11199 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 213432358 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22270041529344 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚479377912976896 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22270275211264 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 208680582 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚457090594582530 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚457090348561878 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22270041529344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚506780360621056 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 518242755 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 518242755 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22206904772470 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚457107871447552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚81054966 _step‚921 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 304784025 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 127625614 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚81054255 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528¦M!Œ'ª-ûM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚457107871447552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚81054966 _step‚921 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 304784025 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 127625614 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚81054255 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚457107889450496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚713 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 518216383 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 208679869 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚711 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚484134386200064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22645688009216 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚479378164661760 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 _runtime‚13270.213954404 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 213432360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚484134362259968 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22270275211264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚479378164661760 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 213432358 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 213432360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 % _timestamp‚1.7884772170283399e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 304810395 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 127625616 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 trainer/global_step‚11199 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 213432358 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22270041529344 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚479377912976896 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22270275211264 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 208680582 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚457090594582530 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚457090348561878 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22270041529344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚506780360621056 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 518242755 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 518242755 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22206904772470 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104™;˶4«-/ _wandbruntime‚13270  _runtime‚13270Ê]Â^¬-Ø .loss/FRt2/detection_loss‚2.381584542508864 1loss/SRqqt4/detection_loss‚2.1275589068730674 /loss/FBt3/detection_loss‚0.5976650819307967 _step‚922 2loss/SRqqt1/assignment_loss‚1.7733000539041854 /loss/FBt2/detection_loss‚1.8396159927050273 &loss/total_loss‚39.272283824109344 _runtime‚13270.215096199 0loss/FRt1/assignment_loss‚2.6321280542761087 2loss/SRqqt2/assignment_loss‚1.4494307569058782 0loss/FBt3/assignment_loss‚0.1427409140601659 /loss/FRt4/detection_loss‚2.8155324356309297 /loss/FRt3/assignment_loss‚1.090500846505165 2loss/SRqqt3/assignment_loss‚0.9501447622030181 0loss/SRqqt2/detection_loss‚2.228285762396726 1loss/SRqqt3/detection_loss‚0.7521328979389782 % _timestamp‚1.7884772170302546e+09 0loss/FRt2/assignment_loss‚2.8049532685961043 /loss/FRt1/detection_loss‚2.6825951244682074 /loss/FBt1/detection_loss‚2.4640473663806914 /loss/FRt4/assignment_loss‚2.981802249776906 1loss/FBt2/assignment_loss‚0.21941897818054484 /loss/FRt3/detection_loss‚1.3979247920762967 /loss/SRqqt1/detection_loss‚2.19824269011214 .loss/FBt4/detection_loss‚2.517986379067103 epoch‚9 trainer/global_step‚11199 2loss/SRqqt4/assignment_loss‚1.0001848626674876 1loss/FBt1/assignment_loss‚0.13044860105143813 1loss/FBt4/assignment_loss‚0.09405850389351306ûÅ æc­-Ý .loss/FBt4/detection_loss‚2.517986379067103 epoch‚9 trainer/global_step‚11199 2loss/SRqqt4/assignment_loss‚1.0001848626674876 1loss/FBt1/assignment_loss‚0.13044860105143813 1loss/FBt4/assignment_loss‚0.09405850389351306 .loss/FRt2/detection_loss‚2.381584542508864 1loss/SRqqt4/detection_loss‚2.1275589068730674 /loss/FBt3/detection_loss‚0.5976650819307967 _step‚922 2loss/SRqqt1/assignment_loss‚1.7733000539041854 /loss/FBt2/detection_loss‚1.8396159927050273 &loss/total_loss‚39.272283824109344 _runtime‚13270.215096199 0loss/FRt1/assignment_loss‚2.6321280542761087 2loss/SRqqt2/assignment_loss‚1.4494307569058782 0loss/FBt3/assignment_loss‚0.1427409140601659 /loss/FRt4/detection_loss‚2.8155324356309297 /loss/FRt3/assignment_loss‚1.090500846505165 2loss/SRqqt3/assignment_loss‚0.9501447622030181 0loss/SRqqt2/detection_loss‚2.228285762396726 1loss/SRqqt3/detection_loss‚0.7521328979389782 % _timestamp‚1.7884772170302546e+09 0loss/FRt2/assignment_loss‚2.8049532685961043 /loss/FRt1/detection_loss‚2.6825951244682074 /loss/FBt1/detection_loss‚2.4640473663806914 /loss/FRt4/assignment_loss‚2.981802249776906 1loss/FBt2/assignment_loss‚0.21941897818054484 /loss/FRt3/detection_loss‚1.3979247920762967 /loss/SRqqt1/detection_loss‚2.19824269011214šÍV>‘®-‚(:† ¥þçÔà˜Þ† gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚138.302' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.5755 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.|Ðì¯-‚(:‚  ¥þçÔû disk.rbd0.in‚0.015625% memory_percent‚41.426730155944824 network.sent‚ 3.014766e+07 disk.sda1.out‚0" disk.sda2.out‚1473.92529296875# proc.memory.rssMB‚ 3032.23828125) proc.memory.percent‚18.50731372833252 cpu‚25.204363235679832 disk.sda.in‚ 5.66796875 disk.dm-0.in‚531780.51171875 disk.md0.in‚ 11.1015625 disk.md0.out‚ 1420.515625) proc.memory.availableMB‚ 9596.64453125 proc.cpu.threads‚17# disk.nvme1n1.out‚76008.78515625 disk.sdb.in‚ 5.43359375 disk.sdb1.in‚0 disk.md0p1.in‚ 11.1015625 disk.sdb2.in‚ 5.43359375 network.recv‚ 4.563049e+06# disk.nvme1n1.in‚201585.86328125! disk.dm-0.out‚156267.83984375 disk.sda1.in‚0" disk.sdb2.out‚1473.92529296875 disk.rbd0.out‚ 3.646484375) disk./.usagePercent‚83.15699421319682 disk.rbd2.in‚0! disk.sdb.out‚1473.92529296875" disk.nvme0n1.out‚ 80259.0546875 disk.sda2.in‚ 5.66796875% disk./.usageGB‚2974.0969047546387 disk.rbd2.out‚ 3.87109375 disk.sdb1.out‚0! disk.sda.out‚1473.92529296875" disk.nvme0n1.in‚330194.6484375 disk.md0p1.out‚ 1420.515625úRðä°-‚(:„  ´þçԃݫ‡ disk.rbd0.in‚0.015625# disk.nvme1n1.in‚201585.86328125" disk.sda2.out‚1476.02001953125 disk.md0.out‚ 1422.54296875 disk.rbd0.out‚ 3.646484375 disk.md0p1.in‚ 11.1015625 network.sent‚ 3.0257238e+07) disk./.usagePercent‚83.14930889521533 disk.md0.in‚ 11.1015625# disk.nvme0n1.out‚80307.98828125 disk.sda1.out‚0( proc.memory.availableMB‚ 9596.6328125 cpu‚25.204170106103685 disk.sdb.in‚ 5.43359375! disk.sdb.out‚1476.02001953125 disk.sda2.in‚ 5.66796875" disk.sdb2.out‚1476.02001953125# proc.memory.rssMB‚ 3032.23828125 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.sda.in‚ 5.66796875 disk.sda1.in‚0) proc.memory.percent‚18.50731372833252 disk.rbd2.in‚0 disk.rbd2.out‚ 3.87109375 disk.sdb1.in‚0! disk.sda.out‚1476.02001953125" disk.nvme0n1.in‚330194.6484375 network.recv‚ 4.570077e+06% disk./.usageGB‚2973.8220405578613 disk.nvme1n1.out‚ 76008.90625! disk.dm-0.out‚156316.89453125 disk.md0p1.out‚ 1422.54296875 disk.sdb2.in‚ 5.43359375% memory_percent‚41.426801681518555 disk.dm-0.in‚531780.51171875+㘱-‚(:… ´þçÔ¬Ú܇ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚135.184' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.796 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$à›²-‚(: ÃþçÔà﵇ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚119.368' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.841999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0AÒ燳-‚(:ü ÃþçÔÈÀêˆ disk.sda1.out‚0 disk.md0p1.in‚ 11.1015625) disk./.usagePercent‚83.15531836291613" disk.nvme0n1.in‚330194.6484375) proc.memory.percent‚18.50731372833252" disk.nvme1n1.out‚ 76009.3671875 disk.md0.in‚ 11.1015625 disk.sdb.in‚ 5.43359375 disk.sdb.out‚1478.3330078125 disk.dm-0.in‚531780.51171875 disk.md0p1.out‚ 1424.7734375 disk.sda2.in‚ 5.66796875 disk.rbd0.out‚ 3.646484375 disk.rbd2.out‚ 3.87109375 disk.sdb2.in‚ 5.43359375 disk.sdb1.in‚0 disk.sda.out‚1478.3330078125% memory_percent‚41.425251960754395) proc.memory.availableMB‚ 9596.88671875 disk.rbd2.in‚0 network.recv‚ 4.573957e+06$ disk./.usageGB‚2974.036968231201! disk.sdb2.out‚1478.3330078125 disk.sda.in‚ 5.66796875 disk.dm-0.out‚156390.2265625 cpu‚25.20397211465743 proc.cpu.threads‚17# disk.nvme1n1.in‚201585.86328125 disk.sdb1.out‚0! disk.nvme0n1.out‚ 80380.859375 disk.sda1.in‚0! disk.sda2.out‚1478.3330078125# proc.memory.rssMB‚ 3032.23828125 network.sent‚ 3.0260358e+07 disk.md0.out‚ 1424.7734375 disk.rbd0.in‚0.015625^»ó¬’´-‚(:‡ ÒþçÔ†ÄÛ… gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚125.603' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.40075 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡{!‡µ-‚(:ü ÒþçÔ˜æù† disk.sda.in‚ 5.66796875 disk.sdb2.in‚ 5.43359375 proc.cpu.threads‚17) disk./.usagePercent‚83.15729691578699 disk.rbd2.in‚0" disk.nvme1n1.out‚ 76009.5234375 disk.sdb.in‚ 5.43359375% memory_percent‚41.426753997802734 cpu‚25.20376913690964 network.sent‚ 3.0266711e+07% disk./.usageGB‚2974.1077308654785 disk.sda2.in‚ 5.66796875 disk.sdb2.out‚1480.212890625" disk.nvme0n1.in‚330194.6484375! disk.dm-0.out‚156603.24609375 disk.md0p1.out‚ 1426.57421875 disk.rbd0.in‚0.015625# proc.memory.rssMB‚ 3032.23828125 network.recv‚ 4.578487e+06 disk.rbd2.out‚ 3.87109375 disk.sda.out‚1480.212890625# disk.nvme1n1.in‚201585.86328125 disk.sdb.out‚1480.212890625 disk.sdb1.in‚0# disk.nvme0n1.out‚80593.72265625 disk.dm-0.in‚531780.51171875 disk.sda1.out‚0' proc.memory.availableMB‚ 9596.640625 disk.sdb1.out‚0 disk.sda2.out‚1480.212890625 disk.md0.in‚ 11.1015625 disk.md0.out‚ 1426.57421875 disk.rbd0.out‚ 3.646484375) proc.memory.percent‚18.50731372833252 disk.sda1.in‚0 disk.md0p1.in‚ 11.1015625rN©4¶-/ _wandbruntime‚13324  _runtime‚13324WýRXË·-Å ' lr-AdamW/pg1‚1.2092748045276386e-05 ' lr-AdamW/pg2‚1.2092748045276386e-05 trainer/global_step‚11249 % _timestamp‚1.7884772717192256e+09 _runtime‚13324.904392339 _step‚923á+~ÿи-Ê ' lr-AdamW/pg1‚1.2092748045276386e-05 ' lr-AdamW/pg2‚1.2092748045276386e-05 trainer/global_step‚11249 % _timestamp‚1.7884772717192256e+09 _runtime‚13324.904392339 _step‚923›•èU4¹-/ _wandbruntime‚13324  _runtime‚13324F¸ä º-ìO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚481399180470272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 214327833 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚81394283 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 306121629 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚459032905263104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚459032887260160 VGDevice™!·«àStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22366526878720 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚481399180470272 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 306121629 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 209549195 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚436722431303184 trainer/global_step‚11249 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚508920349200384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚459025315627176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22743758833152 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚459025561647620 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚13324.905932176 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 520449464 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚626 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 520423118 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚81394909 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚459032887260160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22302884323992 % _timestamp‚1.7884772717204933e+09$w´÷'»-ñO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚459032887260160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22302884323992 % _timestamp‚1.7884772717204933e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚481399180470272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 214327833 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚81394283 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 306121629 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚459032905263104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚459032887260160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22366526878720 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚481399180470272 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 306121629 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 209549195 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚436722431303184 trainer/global_step‚11249 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚508920349200384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚459025315627176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22743758833152 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚459025561647620 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚13324.905932176 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 520449464 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚626 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 520423118 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚81394909 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273œ™L 4¼-/ _wandbruntime‚13326  _runtime‚13326Œfâü&½-öM ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚459072191300096 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚481440682217984 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 520494457 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚486218584702464 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22745773401600 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚459072191300096 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 209566560 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚459065050494238 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 306148347 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 306148347 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22745482795520 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚459064804473586 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚436759978699968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22368257235968 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚459072173297152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 214346110 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 520468085 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 _runtime‚13326.045767451 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 520468085 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22305071794270 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 209567200 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 520494457 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 214346110 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22304842812978 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚508964043557888 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚459072173297152 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 214346108 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 520494457 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚486218584702464 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22745773401600 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚459072191300096 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 209566560 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚459065050494238 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 306148347 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 306148347 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22745482795520 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚459064804473586 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚436759978699968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22368257235968 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚459072173297152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 214346110 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 520468085 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 _runtime‚13326.045767451 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 520468085 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22305071794270 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 209567200 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 520494457 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 214346110 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22304842812978 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚508964043557888 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚459072173297152 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 214346108 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚483461698026496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚460997189109760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 215241583 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚511104694530048 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22464508916736 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚460997189109760 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚461000017559328 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7884773292994034e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚661 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚438598949174384 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22400822364500 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚460997207112704 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚460997207112704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚13382.484444922 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 307459581 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 210439170 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚483461698026496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 128695012 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚511104384164864 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 215241585 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 307459581 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚928 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22464742585344 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚460999771538884 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 128695014 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚81743495 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 522674820 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 210438507 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136¤á™Ï-ñO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚13382.484444922 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 307459581 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 210439170 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚483461698026496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 128695012 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚511104384164864 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 215241585 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 307459581 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚928 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22464742585344 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚460999771538884 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 128695014 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚81743495 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 522674820 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 210438507 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚663 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 522701166 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚488261099234816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 307433237 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚483461698026496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚460997189109760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 215241583 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚511104694530048 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22464508916736 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚460997189109760 YEDeviceStaǃn'Ý tsMonitor.on_train_batch_start/requested_bytes.all.allocated‚461000017559328 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7884773292994034e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚661 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚438598949174384 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22400822364500 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚460997207112704 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚460997207112704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 0}Óž4Ð-/ _wandbruntime‚13383  _runtime‚13383©ï@ü&Ñ-öM ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22466706624512 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚511148378765312 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚511148055830528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 307459929 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚11299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚438636496571168 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚461039506405946 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 215259860 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 307486299 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22466706624512 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚483503199774208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22466472942592 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚438636479531808 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 210455941 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22845595560448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 322934784 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 522719787 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 522746159 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚690 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚81750815 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚688 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 128705814 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22466472942592 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚483503199774208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚929 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 522746159 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22845296565760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 % _timestamp‚1.7884773302615647e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚488302759264768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736¸ØÀ'Ò-ûM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22466706624512 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚511148378765312 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚511148055830528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 307459929 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚11299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚438636496571168 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚461039506405946 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 215259860 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 307486299 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22466706624512 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚483503199774208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22466472942592 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚438636479531808 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 210455941 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22845595560448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 322934784 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 522719787 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 522746159 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚690 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚81750815 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚688 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 128705814 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22466472942592 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚483503199774208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚929 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 522746159 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22845296565760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 % _timestamp‚1.7884773302615647e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚488302759264768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0¡Äoá4Ó-/ _wandbruntime‚13383  _runtime‚13383@ˆý]Ô-× 0loss/FRt1/assignment_loss‚2.7316222353415056 2loss/SRqqt3/assignment_loss‚0.8994001479759971 1loss/SRqqt1/detection_loss‚1.9756052951987197 _step‚930 1loss/FBt2/assignment_loss‚0.09706325394309413 .loss/FRt1/detection_loss‚2.822623863364711 /loss/FBt3/detection_loss‚0.4902534902095795 ,loss/FBt4/detection_loss‚2.5162313580513 0loss/FRt3/assignment_loss‚1.1327777127676373 2loss/SRqqt1/assignment_loss‚1.2224142269753828 2loss/SRqqt2/assignment_loss‚1.2163978373365743 2loss/SRqqt4/assignment_loss‚1.4186330121990882 1loss/FBt1/assignment_loss‚0.14626394604782716 0loss/SRqqt2/detection_loss‚2.423975782735007 /loss/FBt1/detection_loss‚2.6302831172943115 &loss/total_loss‚41.542445510076945 /loss/FRt4/assignment_loss‚2.956768560868043 /loss/FRt4/detection_loss‚3.2878468999495873 1loss/SRqqt3/detection_loss‚0.8557819743502013 _runtime‚13383.449265795 .loss/FRt2/detection_loss‚2.004484016917652 /loss/FRt3/detection_loss‚1.3195986077189445 epoch‚9 % _timestamp‚1.7884773302645683e+09 trainer/global_step‚11299 /loss/FRt2/assignment_loss‚3.115468613612346 0loss/FBt3/assignment_loss‚0.2281063113358573 1loss/FBt4/assignment_loss‚0.06460410887180916 1loss/SRqqt4/detection_loss‚3.2333376972298873 /loss/FBt2/detection_loss‚2.7529034397818823ÇþžbÕ-Ü ,loss/FBt4/detection_loss‚2.5162313580513 0loss/FRt3/assignment_loss‚1.1327777127676373 2loss/SRqqt1/assignment_loss‚1.2224142269753828 2loss/SRqqt2/assignment_loss‚1.2163978373365743 2loss/SRqqt4/assignment_loss‚1.4186330121990882 1loss/FBt1/assignment_loss‚0.14626394604782716 0loss/SRqqt2/detection_loss‚2.423975782735007 /loss/FBt1/detection_loss‚2.6302831172943115 &loss/total_loss‚41.542445510076945 /loss/FRt4/assignment_loss‚2.956768560868043 /loss/FRt4/detection_loss‚3.2878468999495873 1loss/SRqqt3/detection_loss‚0.8557819743502013 _runtime‚13383.449265795 .loss/FRt2/detection_loss‚2.004484016917652 /loss/FRt3/detection_loss‚1.3195986077189445 epoch‚9 % _timestamp‚1.7884773302645683e+09 trainer/global_step‚11299 /loss/FRt2/assignment_loss‚3.115468613612346 0loss/FBt3/assignment_loss‚0.2281063113358573 1loss/FBt4/assignment_loss‚0.06460410887180916 1loss/SRqqt4/detection_loss‚3.2333376972298873 /loss/FBt2/detection_loss‚2.7529034397818823 0loss/FRt1/assignment_loss‚2.7316222353415056 2loss/SRqqt3/assignment_loss‚0.8994001479759971 1loss/SRqqt1/detection_loss‚1.9756052951987197 _step‚930 1loss/FBt2/assignment_loss‚0.09706325394309413 .loss/FRt1/detection_loss‚2.822623863364711 /loss/FBt3/detection_loss‚0.4902534902095795¢ì0’Ö-‚(:‡ ÿçÔ¥‰ð‡ gpu.0.gpu‚27.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚139.251' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.81275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôv×-‚(:ë ÿçÔë¯ßˆ network.recv‚ 4.602817e+06 disk.rbd0.out‚ 3.646484375 disk.rbd2.out‚ 3.87109375# disk.nvme1n1.in‚201585.86328125) proc.memory.percent‚18.50731372833252" disk.nvme0n1.in‚330194.6484375 disk.dm-0.in‚531780.51171875 disk.sda1.in‚0 disk.md0.out‚ 1435.3125# disk.nvme1n1.out‚76010.68359375 disk.sda.out‚ 1489.3046875 proc.cpu.threads‚17( disk./.usagePercent‚83.1775839623176 disk.sda2.out‚ 1489.3046875 disk.md0p1.out‚ 1435.3125 disk.rbd2.in‚0 disk.sdb2.out‚ 1489.3046875 disk.md0.in‚ 11.1015625 disk.rbd0.in‚0.015625 disk.sdb1.out‚0$ memory_percent‚41.42584800720215( proc.memory.availableMB‚ 9596.7890625 disk.md0p1.in‚ 11.1015625 disk.sdb1.in‚0 disk.sda.in‚ 5.66796875" disk.nvme0n1.out‚ 81572.8203125 disk.sdb2.in‚ 5.43359375# proc.memory.rssMB‚ 3032.23828125 cpu‚25.202847398178484 disk.sdb.in‚ 5.43359375 disk.sdb.out‚ 1489.3046875 network.sent‚ 3.0410764e+07$ disk./.usageGB‚2974.833293914795! disk.dm-0.out‚157583.50390625 disk.sda1.out‚0 disk.sda2.in‚ 5.66796875IÈÞ’Ø-‚(:‡ ¬ÿçÔîÎ… gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚143.595' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.89875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õB‡Ù-‚(:ü ¬ÿçÔá’ò‰# proc.memory.rssMB‚ 3032.23828125 proc.cpu.threads‚17 network.sent‚ 3.0505615e+07" disk.nvme0n1.in‚330194.6484375 disk.dm-0.in‚531780.51171875 disk.sdb2.in‚ 5.43359375" disk.sdb2.out‚1490.94189453125 disk.rbd0.out‚ 3.646484375 disk.dm-0.out‚157608.2109375 disk.md0p1.in‚ 11.1015625 disk.md0p1.out‚ 1436.88671875 disk.rbd2.in‚0$ disk./.usageGB‚2974.789390563965 disk.sdb.in‚ 5.43359375 disk.sdb1.out‚0 disk.rbd0.in‚0.015625& proc.memory.availableMB‚ 9596.53125) proc.memory.percent‚18.50731372833252) disk./.usagePercent‚83.17635640625342# disk.nvme1n1.in‚201585.86328125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚ 5.66796875 disk.rbd2.out‚ 3.87109375! disk.sda.out‚1490.94189453125 disk.nvme0n1.out‚ 81596.875! disk.sdb.out‚1490.94189453125 disk.sdb1.in‚0 disk.sda.in‚ 5.66796875" disk.sda2.out‚1490.94189453125 cpu‚25.202653776854362 network.recv‚ 4.610244e+06" disk.nvme1n1.out‚ 76011.3359375 disk.md0.in‚ 11.1015625 disk.md0.out‚ 1436.88671875$ memory_percent‚41.42742156982422iƒA÷ŽÚ-‚(:ƒ »ÿçÔ‰÷á… gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚137.36' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.34 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ù<(„Û-‚(:ù »ÿçÔ·µŽˆ disk.sdb1.in‚0 disk.sda.in‚ 5.66796875 disk.md0p1.out‚ 1439.06640625 disk.sdb2.in‚ 5.43359375# proc.memory.rssMB‚ 3032.23828125 disk.rbd2.in‚0" disk.nvme0n1.out‚ 81741.0703125 disk.sda1.out‚0 disk.md0.in‚ 11.1015625) proc.memory.percent‚18.50731372833252) disk./.usagePercent‚83.17828536197683 disk.dm-0.in‚531780.51171875) proc.memory.availableMB‚ 9596.91015625 cpu‚25.2024906627014% disk./.usageGB‚2974.8583793640137 disk.sdb.out‚ 1493.19921875 disk.sda1.in‚0 disk.sda2.in‚ 5.66796875 disk.md0p1.in‚ 11.1015625 disk.rbd2.out‚ 3.87109375 network.sent‚ 3.0513089e+07 disk.sdb.in‚ 5.43359375! disk.dm-0.out‚157821.10546875# disk.nvme1n1.in‚201585.86328125 disk.sdb1.out‚0 disk.sda2.out‚ 1493.19921875 disk.md0.out‚ 1439.06640625 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 3.646484375# disk.nvme1n1.out‚76080.03515625% memory_percent‚41.425108909606934 proc.cpu.threads‚17 network.recv‚ 4.613854e+06 disk.sda.out‚ 1493.19921875" disk.nvme0n1.in‚330194.6484375 disk.sdb2.out‚ 1493.19921875–RÖ4Ü-/ _wandbruntime‚13439  _runtime‚13439±h‹ÉÝ-à & lr-AdamW/pg1‚9.398252828289353e-06 & lr-AdamW/pg2‚9.398252828289353e-06 trainer/global_step‚11349 % _timestamp‚1.7884773864427636e+09 _runtime‚13439.627669063 _step‚931j܈ÏÎÞ-È & lr-AdamW/pg2‚9.398252828289353e-06 trainer/global_step‚11349 % _timestamp‚1.7884773864427636e+09 _runtime‚13439.627669063 _step‚931 & lr-AdamW/pg1‚9.398252828289353e-06£b`S4ß-/ _wandbruntime‚13439  _runtime‚13439¸Þú¡Oà-ìO _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22498989386092 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 308797533 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚462961490959360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 211325410 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚82090296 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚2à‘YX£6344 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 129235112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 308771189 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22562724623360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚485524215582720 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 211324707 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚440475467045584 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22562958291968 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚485524215582720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22498760405008 % _timestamp‚1.7884773864437952e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 524926522 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22562958291968 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 308797533 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚703 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22943220455936 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22562724623360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 216155335 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 524952868 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 129235114 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 327142400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _step‚932 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚462961490959360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚701 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22943523658240 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 308771189 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 216155335 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚513288821395456 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚462974473471036 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚440475484084944 _runtime‚13439.628823308 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚485524467254272 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚513288494253056 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 216155333 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚82089595ÉëÔz÷'á-ñO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 308771189 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22562724623360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚485524215582720 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 211324707 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚440475467045584 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22562958291968 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚485524215582720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22498760405008 % _timestamp‚1.7884773864437952e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 524926522 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22562958291968 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 308797533 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚703 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22943220455936 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22562724623360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 216155335 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 524952868 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 129235114 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 327142400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _step‚932 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚462961490959360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚701 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22943523658240 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 308771189 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 216155335 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚513288821395456 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚462974473471036 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚440475484084944 _runtime‚13439.628823308 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚485524467254272 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚513288494253056 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 216155333 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚82089595 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22498989386092 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 308797533 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚462961490959360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 211325410 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚82090296 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 129235112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0¤Úþß×’â-‚(:‡ ÊÿçÔðÁý… gpu.0.gpu‚22.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚141.427' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.35675 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_üt‘‰ã-‚(:þ ÊÿçÔ¦ŽŒ‡ disk.sdb1.in‚0 disk.rbd2.out‚ 3.87109375 disk.rbd0.in‚0.015625 disk.rbd2.in‚0) disk./.usagePercent‚83.18178862715438$ disk./.usageGB‚2974.983673095703 disk.sdb.in‚ 5.43359375 disk.sdb1.out‚0 disk.sda1.in‚0 disk.md0p1.in‚ 11.1015625 network.recv‚ 4.618623e+06 disk.sda.in‚ 5.66796875! disk.sdb2.out‚1494.9052734375# disk.nvme1n1.in‚201585.86328125" disk.nvme1n1.out‚ 76107.6640625 cpu‚25.20230623378485 proc.cpu.threads‚17" disk.nvme0n1.in‚330194.6484375# proc.memory.rssMB‚ 3032.23828125 disk.sda.out‚1494.9052734375# disk.nvme0n1.out‚81885.10546875 disk.sda2.in‚ 5.66796875$ memory_percent‚41.42882823944092) proc.memory.percent‚18.50731372833252 disk.rbd0.out‚ 3.646484375 disk.md0.out‚ 1440.7109375 disk.dm-0.in‚531780.51171875 network.sent‚ 3.0517373e+07! disk.dm-0.out‚157992.76953125 disk.sda1.out‚0! disk.sda2.out‚1494.9052734375 disk.md0p1.out‚ 1440.7109375 disk.sdb2.in‚ 5.43359375 disk.md0.in‚ 11.1015625) proc.memory.availableMB‚ 9596.30078125 disk.sdb.out‚1494.9052734375`æ4ä-/ _wandbruntime‚13440  _runtime‚13440¢DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 211342473 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚82096559 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 216173608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚513332172430336 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 524997861 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 524997861 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚463013716297002 trainer/global_step‚11349 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚463000794999296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22564688649216 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 524971489 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚463000776996352 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 308797881 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 129245914 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 % _timestamp‚1.7884773874697862e+09 _step‚933 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22945214663168 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚440513014442368 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚490386957767168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 211343171 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚463013962317654 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚82097255 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚490386981707264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 308824251 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚513332491170816 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 _runtime‚13440.655573753 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 216173610 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚485565465645568 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚696 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22500947875286 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 216173610 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22500718893994 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 211342473 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚82096559 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 216173608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚513332172430336 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 524997861 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 524997861 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚463013716297002 trainer/global_step‚11349 QBDeviceStatsMonitor.oâ¤ê•n_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚463000794999296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22564688649216 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 524971489 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚463000776996352 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 308797881 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 129245914 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 % _timestamp‚1.7884773874697862e+09 _step‚933 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22945214663168 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚440513014442368 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚490386957767168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 211343171 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚463013962317654 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚82097255 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚490386981707264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 308824251 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚513332491170816 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 _runtime‚13440.655573753 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 216173610 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚485565465645568 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚696 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22500947875286 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 216173610 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22500718893994 F+î’ï-‚(:‡ ÷ÿçÔ¬Œ§ˆ gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚134.575' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.64375 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÎejÝ4ð-/ _wandbruntime‚13497  _runtime‚13497œ-Èñ- $ _timestamp‚1.788477443846884e+09 _runtime‚13497.031858758 _step‚935 & lr-AdamW/pg1‚8.176792354982066e-06 & lr-AdamW/pg2‚8.176792354982066e-06 trainer/global_step‚11399 IÍò-Ç & lr-AdamW/pg2‚8.176792354982066e-06 trainer/global_step‚11399 $ _timestamp‚1.788477443846884e+09 _runtime‚13497.031858758 _step‚935 & lr-AdamW/pg1‚8.176792354982066e-06§:wXX4ó-/ _wandbruntime‚13497  _runtime‚13497ûÎòñ'ô-ëO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚492429496239616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 217069085 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 212211569 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚492429472299520 trainer/global_step‚11399 _runtime‚13497.032686392 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 212212243 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22660940329984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚487586733138944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚464925810811904 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚464925792808960 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22660940329984 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 310109141 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23043382973440 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23043096548352 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚464948683362300 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚442351984916784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚464925810811904 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 129775212 _step‚936 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 217069083 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚487586984810496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚672 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚515472879213056 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚487586733138944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22661173998592 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 527204570 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 $ _timestamp‚1.788477443847654e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 217069085 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 129775214 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚82437029 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚464948929382744 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22661173998592 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 310135485ì 5>Yõ-ðO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 217069085 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 212211569 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚492429472299520 trainer/global_step‚11399 _runtime‚13497.032686392 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 212212243 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22660940329984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚487586733138944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚464925810811904 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚464925792808960 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22660940329984 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 310109141 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23043382973440 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23043096548352 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚464948683362300 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚442351984916784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚464925810811904 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 129775212 _step‚936 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 217069083 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚487586984810496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚672 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚515472879213056 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚487586733138944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22661173998592 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 527204570 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 $ _timestamp‚1.788477443847654e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 217069085 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 129775214 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚82437029 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚464948929382744 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22661173998592 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 310135485 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚492429496239616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360¨ØÔÕÐ4ö-/ _wandbruntime‚13498  _runtime‚13498ÃM„Gû&÷-õM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚464965096848896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 217087358 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 129786014 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚487628234886656 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚464965096848896 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 217087360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22662904355840 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 310135833 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚515516560515584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚464965078845952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚487627983201792 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 129786016 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚492471156269568 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚442389515274208 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚492471180209664 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23045083408384 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚688 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚82443893 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚487628234886656 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 527249563 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚13498.160900443 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 _step‚937 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 310162203 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚464988418229362 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22598656934502 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22662904355840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚82443207 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 212229221 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 310162203 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22663138037760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22663138037760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚686 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚464988172208710 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚442389532313568 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚464965078845952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 527223191 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 -´ 'ø-úM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 527223191 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚464988172208710 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚442389532313568 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚464965078845952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 527223191 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚464965096848896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 217087358 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 129786014 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚487628234886656 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚464965096848896 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 217087360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22662904355840 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 310135833 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚515516560515584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚464965078845952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚487627983201792 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 129786016 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚492471156269568 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚442389515274208 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚492471180209664 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23045083408384 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚688 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚82443893 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚487628234886656 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 527249563 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚13498.160900443 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 _step‚937 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 310162203 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚464988418229362 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22598656934502 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22662904355840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚82443207 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 212229221 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 310162203 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22663138037760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22663138037760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚686 P7Ìÿ-‚(:ô •€èÔÇÇœ‡ network.recv‚ 4.647721e+06) disk./.usagePercent‚83.17758278905175 disk.rbd2.in‚0# disk.nvme1n1.in‚201585.86328125 disk.sda.out‚1503.5908203125" disk.nvme0n1.in‚330194.6484375 disk.sda1.out‚0 disk.md0p1.in‚ 11.953125 disk.rbd0.in‚0.015625 disk.sdb.in‚5.796875 disk.sdb.out‚1503.5908203125 disk.sdb1.in‚0 disk.sda.in‚6.15625 cpu‚25.201278082681096 disk.sda2.in‚6.15625 disk.md0p1.out‚ 1449.05859375 disk.rbd0.out‚ 3.646484375 disk.sdb1.out‚0$ disk./.usageGB‚2974.833251953125 disk.sda1.in‚0! disk.sda2.out‚1503.5908203125$ memory_percent‚41.42804145812988 proc.cpu.threads‚17 disk.rbd2.out‚ 3.87109375" disk.nvme0n1.out‚ 82461.7265625 disk.sdb2.in‚5.796875! disk.sdb2.out‚1503.5908203125 disk.md0.in‚ 11.953125! disk.dm-0.out‚159082.48046875( proc.memory.availableMB‚ 9596.4296875# proc.memory.rssMB‚ 3032.23828125) proc.memory.percent‚18.50731372833252 network.sent‚ 3.0829312e+07 disk.md0.out‚ 1449.05859375# disk.nvme1n1.out‚76620.75390625 disk.dm-0.in‚531780.51171875/_ÛÝ€.‚(:… ¤€èÔܱé‡ gpu.0.gpu‚46.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚135.184' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.796 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€«×….‚(:ú ¤€èÔâàˆ disk.rbd2.in‚0! disk.sda.out‚1505.61865234375# proc.memory.rssMB‚ 3032.23828125 network.recv‚ 4.651958e+06 disk.rbd0.in‚0.015625 disk.sda1.out‚0 disk.sda2.in‚6.15625 proc.cpu.threads‚17) disk./.usagePercent‚83.17820909969636 disk.sdb2.in‚5.796875 disk.rbd0.out‚ 3.646484375 disk.rbd2.out‚ 3.87109375 disk.md0p1.in‚ 11.953125 disk.md0.in‚ 11.953125 disk.sdb.in‚5.796875! disk.sdb.out‚1505.61865234375 disk.md0p1.out‚ 1451.03515625% disk./.usageGB‚2974.8556518554688# disk.nvme1n1.in‚201585.86328125" disk.nvme0n1.in‚330194.6484375 disk.dm-0.in‚531780.51171875 disk.sda1.in‚0) proc.memory.percent‚18.50731372833252 cpu‚25.201086456173613 network.sent‚ 3.0837473e+07 disk.sdb1.out‚0# disk.nvme0n1.out‚82589.84765625" disk.sda2.out‚1505.61865234375) proc.memory.availableMB‚ 9597.55078125 disk.md0.out‚ 1451.03515625" disk.nvme1n1.out‚ 76621.4296875! disk.dm-0.out‚159211.27734375$ memory_percent‚41.42119884490967" disk.sdb2.out‚1505.61865234375 disk.sdb1.in‚0 disk.sda.in‚6.15625ðÆhÝ’‚.‚(:‡ ³€èÔ–‰à… gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚141.087' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.27175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|±æ’yƒ.‚(:î ³€èÔÛŸð† disk.dm-0.in‚531780.51171875 disk.sdb2.in‚5.796875 disk.sdb1.in‚0! disk.sda.out‚1507.35302734375! disk.dm-0.out‚159298.16796875 proc.cpu.threads‚17) disk./.usagePercent‚83.17913491311518% disk./.usageGB‚2974.8887634277344 disk.md0.out‚ 1452.71875 disk.rbd2.out‚ 3.87109375 disk.sda1.in‚0 disk.sda1.out‚0# disk.nvme1n1.in‚201585.86328125 disk.sda2.in‚6.15625% proc.memory.availableMB‚ 9597.4375 network.sent‚ 3.0840569e+07 disk.rbd0.in‚0.015625 disk.rbd2.in‚0$ memory_percent‚41.42189025878906 cpu‚25.2008163114386" disk.sdb2.out‚1507.35302734375 disk.sdb1.out‚0 network.recv‚ 4.655835e+06 disk.md0p1.in‚ 11.953125 disk.sda.in‚6.15625" disk.sda2.out‚1507.35302734375 disk.md0.in‚ 11.953125" disk.nvme1n1.out‚ 76621.5859375# proc.memory.rssMB‚ 3032.23828125) proc.memory.percent‚18.50731372833252 disk.md0p1.out‚ 1452.71875 disk.rbd0.out‚ 3.646484375 disk.sdb.in‚5.796875! disk.sdb.out‚1507.35302734375" disk.nvme0n1.in‚330194.6484375# disk.nvme0n1.out‚82676.58203125'lL4„./ _wandbruntime‚13554  _runtime‚13554ÐÀõ”Ç….Á % lr-AdamW/pg1‚7.03953520410753e-06 % lr-AdamW/pg2‚7.03953520410753e-06 trainer/global_step‚11449 % _timestamp‚1.7884775016186142e+09 _runtime‚13554.803544682 _step‚939|Ѵ̆.Æ trainer/global_step‚11449 % _timestamp‚1.7884775016186142e+09 _runtime‚13554.803544682 _step‚939 % lr-AdamW/pg1‚7.03953520410753e-06 % lr-AdamW/pg2‚7.03953520410753e-06«ñ5^É4‡./ _wandbruntime‚13554  _runtime‚13554ðiŸ„ˆ.ìO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 311447093 ZKDeviceStatsMonitor.on_train_batch_start/resŒÏn'erved_bytes.large_pool.allocated‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 130315314 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚466923385294452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 130315312 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22759156036608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚82782530 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚494513670801920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 217982833 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚444228519827344 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22759389705216 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 529456272 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚489649250695168 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22694865467108 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7884775016193812e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22759156036608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚489649250695168 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚517656880779776 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚466923139274008 _step‚940 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚489649502366720 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 217982835 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23143186037760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚489649502366720 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚466890112661504 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22759389705216 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚466890094658560 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 _runtime‚13554.804469975 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚466890094658560 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 311473437 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 217982833 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚444228519827344 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22759389705216 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 529456272 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚489649250695168 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22694865467108 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7884775016193812e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22759156036608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚489649250695168 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚517656880779776 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚466923139274008 _step‚940 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚489649502366720 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 217982835 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23143186037760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚489649502366720 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚466890112661504 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22759389705216 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚466890094658560 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 _runtime‚13554.804469975 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚466890094658560 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 311473437 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚466923385294452¬y4æ’4Š./ _wandbruntime‚13555  _runtime‚13555k³Ñü&‹.öM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚466929398698496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 311473785 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚654 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884775026357641e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚466929398698496 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚652 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22761353744384 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚466962874141070 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22696823956302 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 218001110 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23144894777856 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 311500155 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚82789607 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚444266033145408 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚489690752442880 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23145176995328 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22761120062464 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 529501265 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚489690500758016 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 311473785 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 529501265 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚494555378712064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 218001108 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 213115721 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚13555.822179607 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 130326114 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚82790259 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22696594975010 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚489690752442880 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚494555354771968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚941 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22761353744384 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 130326116 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚517700249549824 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 529474893 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚444266050184768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22761120062464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚517700555707392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0Bñ”H Œ.ûM EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 529501265 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚489690500758016 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 311473785 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 529501265 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚494555378712064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 218001108 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 213115721 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚13555.822179607 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 130326114 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚82790259 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22696594975010 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚489690752442880 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚494555354771968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚941 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22761353744384 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 130326116 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚517700249549824 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 529474893 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚444266050184768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22761120062464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚517700555707392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚466929398698496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 311473785 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚654 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884775026357641e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚466929398698496 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚652 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22761353744384 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚466962874141070 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22696823956302 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 218001110 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23144894777856 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 311500155 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚82789607 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚444266033145408 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚489690752442880 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23145176995328 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22761120062464 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0­Cä͉4./ _wandbruntime‚13555  _runtime‚13555Š;2¶]Ž.× 0loss/FRt1/assignment_loss‚3.0433059392436856 0loss/FRt4/assignment_loss‚3.4229762421713934 _runtime‚13555.823567622 1loss/SRqqt4/detection_loss‚3.0223667590241683 0loss/FBt3/detection_loss‚0.44993525879808527 _step‚942 0loss/FRt3/assignment_loss‚1.1154843607886893 1loss/SRqqt2/assignment_loss‚1.511901736492291 0loss/SRqqt1/detection_loss‚2.267398476600647 -loss/FBt2/detection_loss‚2.28487497142383 epoch‚9 2loss/SRqqt1/assignment_loss‚1.6246698000713398 /loss/FRt4/detection_loss‚2.7715955326954522 /loss/FBt4/detection_loss‚2.7162812600533166 /loss/FRt2/assignment_loss‚2.931639266949074 1loss/SRqqt2/detection_loss‚2.1009140126407146 trainer/global_step‚11449 % _timestamp‚1.7884775026386983e+09 1loss/FBt1/assignment_loss‚0.07985061236044763 1loss/FBt3/assignment_loss‚0.08058843063842497 0loss/FBt4/assignment_loss‚0.3069675313890888 /loss/FRt1/detection_loss‚2.6891930891621496 /loss/FRt2/detection_loss‚2.0359996254190658 %loss/total_loss‚43.05401473017853 2loss/SRqqt4/assignment_loss‚2.7525335717639936 /loss/FRt3/detection_loss‚1.5586842516528778 -loss/FBt1/detection_loss‚2.38852812235172 2loss/SRqqt3/assignment_loss‚0.8770256524987738 0loss/FBt2/assignment_loss‚0.1640833416254921 1loss/SRqqt3/detection_loss‚0.8572168843638078z·b.Ü 2loss/SRqqt4/assignment_loss‚2.7525335717639936 /loss/FRt3/detection_loss‚1.5586842516528778 -loss/FBt1/detection_loss‚2.38852812235172 2loss/SRqqt3/assignment_loss‚0.8770256524987738 0loss/FBt2/assignment_loss‚0.1640833416254921 1loss/SRqqt3/detection_loss‚0.8572168843638078 0loss/FRt1/assignment_loss‚3.0433059392436856 0loss/FRt4/assignment_loss‚3.4229762421713934 _runtime‚13555.823567622 1loss/SRqqt4/detection_loss‚3.0223667590241683 0loss/FBt3/detection_loss‚0.44993525879808527 _step‚942 0loss/FRt3/assignment_loss‚1.1154843607886893 1loss/SRqqt2/assignment_loss‚1.511901736492291 0loss/SRqqt1/detection_loss‚2.267398476600647 -loss/FBt2/detection_loss‚2.28487497142383 epoch‚9 2loss/SRqqt1/assignment_loss‚1.6246698000713398 /loss/FRt4/detection_loss‚2.7715955326954522 /loss/FBt4/detection_loss‚2.7162812600533166 /loss/FRt2/assignment_loss‚2.931639266949074 1loss/SRqqt2/detection_loss‚2.1009140126407146 trainer/global_step‚11449 % _timestamp‚1.7884775026386983e+09 1loss/FBt1/assignment_loss‚0.07985061236044763 1loss/FBt3/assignment_loss‚0.08058843063842497 0loss/FBt4/assignment_loss‚0.3069675313890888 /loss/FRt1/detection_loss‚2.6891930891621496 /loss/FRt2/detection_loss‚2.0359996254190658 %loss/total_loss‚43.05401473017853®…‚•}.‚(:… €èÔÞ—‡ gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚128.056' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.014 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0q<öf„‘.‚(:ù €èÔÍ·¯ˆ disk.rbd2.out‚ 3.87109375# disk.nvme1n1.in‚201585.86328125! disk.sda.out‚1509.62744140625" disk.sda2.out‚1509.62744140625 network.recv‚ 4.660605e+06) disk./.usagePercent‚83.17347859843623 disk.rbd0.in‚0.015625 disk.sdb1.out‚0# proc.memory.rssMB‚ 3032.23828125$ disk./.usageGB‚2974.686466217041" disk.sdb2.out‚1509.62744140625" disk.nvme0n1.in‚330194.6484375" disk.nvme0n1.out‚ 82948.0390625 disk.sda2.in‚6.15625) proc.memory.percent‚18.50731372833252 disk.md0p1.out‚ 1454.92578125! disk.dm-0.out‚159569.79296875 proc.cpu.threads‚17 disk.md0.out‚ 1454.92578125 disk.dm-0.in‚531780.51171875 disk.sda1.out‚0% memory_percent‚41.391634941101074 cpu‚25.20064416768848 disk.sda1.in‚0 disk.sdb2.in‚5.796875 disk.rbd2.in‚0 disk.sdb.in‚5.796875 disk.sdb1.in‚0 network.sent‚ 3.0847041e+07 disk.md0.in‚ 11.953125 disk.rbd0.out‚ 3.646484375 disk.sda.in‚6.15625) proc.memory.availableMB‚ 9602.39453125 disk.md0p1.in‚ 11.953125# disk.nvme1n1.out‚76621.75390625! disk.sdb.out‚1509.62744140625q$üÐ’.‚(:… Ñ€èÔé ”† gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚134.908' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.727 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÅYÀØ}“.‚(:ò Ñ€èÔä½Â‡ disk.sda1.out‚0" disk.sda2.out‚1511.44775390625% disk./.usageGB‚2974.8114471435547 disk.sdb2.in‚5.796875 disk.md0.out‚ 1456.671875 disk.sdb.in‚5.796875! disk.sdb.out‚1511.44775390625# disk.nvme0n1.out‚83252.46484375$ memory_percent‚41.39230251312256 disk.rbd0.out‚ 3.646484375! disk.sda.out‚1511.44775390625) proc.memory.availableMB‚ 9602.28515625" disk.sdb2.out‚1511.44775390625 disk.rbd0.in‚0.015625 disk.nvme1n1.out‚ 76622.1875 disk.sda.in‚6.15625 disk.sda1.in‚0 disk.md0p1.in‚ 11.953125 disk.md0p1.out‚ 1456.671875 network.sent‚ 3.0954534e+07 network.recv‚ 4.667704e+06) disk./.usagePercent‚83.17697311745013" disk.nvme0n1.in‚330194.6484375 disk.sda2.in‚6.15625 disk.rbd2.in‚0! disk.dm-0.out‚159874.65234375# proc.memory.rssMB‚ 3032.23828125 disk.md0.in‚ 11.953125 disk.rbd2.out‚ 3.87109375# disk.nvme1n1.in‚201585.86328125) proc.memory.percent‚18.50731372833252 cpu‚25.20046300051028 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.dm-0.in‚531780.51171875Þ®¬’”.‚(:‡ à€èÔÍ«Ö‡ gpu.0.gpu‚10.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚142.653' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.66325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ï“ØOu•.‚(:ê à€èÔ»ýë‡ network.recv‚ 4.671756e+06) disk./.usagePercent‚83.17191730156705 disk.sdb.out‚1513.154296875 disk.sda.out‚1513.154296875" disk.nvme0n1.in‚330194.6484375 disk.sdb2.out‚1513.154296875 disk.md0.in‚ 11.953125 disk.rbd2.out‚ 3.87109375# disk.nvme1n1.in‚201585.86328125" disk.nvme1n1.out‚ 76622.3046875$ disk./.usageGB‚2974.630626678467 disk.rbd0.out‚ 3.646484375 disk.sdb.in‚5.796875 disk.sda2.in‚6.15625 disk.sda.in‚6.15625 disk.sda1.out‚0 disk.sda2.out‚1513.154296875( proc.memory.availableMB‚ 9602.4140625# proc.memory.rssMB‚ 3032.23828125) proc.memory.percent‚18.50731372833252 disk.md0.out‚ 1458.30078125 disk.rbd0.in‚0.015625 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.dm-0.out‚159913.8671875 proc.cpu.threads‚17 network.sent‚ 3.0960538e+07 disk.md0p1.in‚ 11.953125 disk.sdb2.in‚5.796875 disk.sdb1.in‚0 disk.nvme0n1.out‚ 83291.5625 cpu‚25.20028117850424 disk.dm-0.in‚531780.51171875 disk.sda1.in‚0 disk.md0p1.out‚ 1458.30078125$ memory_percent‚41.39151573181152±µc¯’–.‚(:‡ ï€èÔ’Ÿ—‡ gpu.0.gpu‚23.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚125.603' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.40075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡˜?Xs—.‚(:è ï€èÔرê‡ disk.sda1.in‚0# proc.memory.rssMB‚ 3032.23828125 disk.md0p1.in‚ 11.953125 disk.sdb2.out‚1515.142578125 disk.md0.out‚ 1460.2265625 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.sdb.out‚1515.142578125 network.recv‚ 4.675506e+06 disk.sda1.out‚0 disk.sdb2.in‚5.796875 disk.rbd0.in‚0.015625! disk.nvme1n1.out‚ 76622.828125$ memory_percent‚41.39304161071777( proc.memory.availableMB‚ 9602.1640625 disk.dm-0.out‚ 160113.84375 disk.sdb1.in‚0 disk.sda.in‚6.15625) proc.memory.percent‚18.50731372833252% disk./.usageGB‚2973.7725868225098 disk.sda2.in‚6.15625" disk.nvme0n1.in‚330194.6484375 disk.md0.in‚ 11.953125 disk.rbd0.out‚ 3.646484375 disk.sdb.in‚5.796875 cpu‚25.200094125276763 disk.sdb1.out‚0 disk.sda.out‚1515.142578125( disk./.usagePercent‚83.1479261480769 disk.md0p1.out‚ 1460.2265625 disk.rbd2.out‚ 3.87109375! disk.nvme0n1.out‚ 83491.015625 network.sent‚ 3.0968088e+07 disk.dm-0.in‚531780.51171875 disk.sda2.out‚1515.142578125# disk.nvme1n1.in‚201585.86328125U0‰€4˜./ _wandbruntime‚13607  _runtime‚13607M'ÜÏÊ™.Ä ' lr-AdamW/pg1‚5.9867183783513935e-06 ' lr-AdamW/pg2‚5.9867183783513935e-06 trainer/global_step‚11499 $ _timestamp‚1.788477554632131e+09 _runtime‚13607.817021012 _step‚943>…&eÏš.É ' lr-AdamW/pg1‚5.9867183783513935e-06 ' lr-AdamW/pg2‚5.9867183783513935e-06 trainer/global_step‚11499 $ _timestamp‚1.788477554632131e+09 _runtime‚13607.817021012 _step‚943¯¡"»4›./ _wandbruntime‚13607  _runtime‚13607(l…#ò'œ.ìO _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22792803507616 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22857605411840 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚650 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 218896585 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22857371743232 trainer/global_step‚11499 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚468854414511104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 531707974 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚944 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 130855412 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 130855414 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚496597893244416 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚496597869304320 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 312811389 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 218896585 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚648 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚491711768251392 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚446105037698544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚468897841206160 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 218896583 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 213982158 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚491711768251392 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚468854396508160 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 312811389 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚83126096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 312785045 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚468854396508160 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22857371743232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2uI€¶.ñO % _timestamp‚1.7884775546331131e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 213982158 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚491711768251392 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚468854396508160 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 312811389 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚83126096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 312785045 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚468854396508160 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22857371743232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22792803507616 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22857605411840 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚650 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 218896585 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22857371743232 trainer/global_step‚11499 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚mçU+A468854414511104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 531707974 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚944 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 130855412 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 130855414 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚496597893244416 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚496597869304320 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 312811389 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 218896585 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚648 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚491711768251392 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚446105037698544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚468897841206160 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 218896583 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 213998996 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚468937084032126 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22859569451008 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚468893700548096 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚496639577214464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 531726595 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 218914858 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 531726595 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 531752967 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚83133456 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22859335769088 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22859335769088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚496639553274368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 218914860 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 312838107 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22859569451008 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 312838107 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23245126749184 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚11499 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608øIöÎÿ& .ùM H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 531726595 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 218914858 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 531726595 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 531752967 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚83133456 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22859335769088 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22859335769088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚496639553274368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 218914860 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 312838107 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22859569451008 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 312838107 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23245126749184 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚11499 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 312811737 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚468937330052778 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 531752967 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚491753269999104 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚468893682545152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚491753018314240 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22794533015518 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 213998996 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚468937084032126 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22859569451008 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚468893700548096 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚496639577214464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736±©ÐCO4¡./ _wandbruntime‚13608  _runtime‚13608´ù÷¬[¢.Õ 0loss/FRt2/assignment_loss‚2.6995407056808474 /loss/FRt3/detection_loss‚1.3504966234262936 0loss/SRqqt1/detection_loss‚2.048320030172666 /loss/FBt3/detection_loss‚0.7085822801394709 _runtime‚13608.773348807 2loss/FBt4/assignment_loss‚0.018984048161605215 %loss/total_loss‚39.81766299366326 0loss/FRt1/assignment_loss‚2.4738003669715507 1loss/FBt3/assignment_loss‚0.02540660327971647 0loss/SRqqt2/detection_loss‚2.205813929438591 1loss/SRqqt4/detection_loss‚2.7370397514767117 .loss/FBt1/detection_loss‚1.903467671945691 .loss/FBt4/detection_loss‚3.045921194553375 1loss/FBt1/assignment_loss‚0.08188762918139147 /loss/FRt2/detection_loss‚2.1864586199773477 1loss/SRqqt3/detection_loss‚0.7618946262469916 /loss/FRt4/detection_loss‚3.3004984829736794 epoch‚9 % _timestamp‚1.7884775555886328e+09 _step‚946 0loss/FRt3/assignment_loss‚1.3382523151544423 /loss/FRt4/assignment_loss‚2.645722783130148 1loss/SRqqt3/assignment_loss‚1.041950934621773 1loss/FBt2/assignment_loss‚0.12284717746464716 trainer/global_step‚11499 1loss/SRqqt1/assignment_loss‚1.234865449136123 /loss/FBt2/detection_loss‚2.0760089479959927 1loss/SRqqt2/assignment_loss‚1.664346009012661 2loss/SRqqt4/assignment_loss‚1.4506395315682445 ,loss/FRt1/detection_loss‚2.6949172819533¤ü¾¡`£.Ú epoch‚9 % _timestamp‚1.7884775555886328e+09 _step‚946 0loss/FRt3/assignment_loss‚1.3382523151544423 /loss/FRt4/assignment_loss‚2.645722783130148 1loss/SRqqt3/assignment_loss‚1.041950934621773 1loss/FBt2/assignment_loss‚0.12284717746464716 trainer/global_step‚11499 1loss/SRqqt1/assignment_loss‚1.234865449136123 /loss/FBt2/detection_loss‚2.0760089479959927 1loss/SRqqt2/assignment_loss‚1.664346009012661 2loss/SRqqt4/assignment_loss‚1.4506395315682445 ,loss/FRt1/detection_loss‚2.6949172819533 0loss/FRt2/assignment_loss‚2.6995407056808474 /loss/FRt3/detection_loss‚1.3504966234262936 0loss/SRqqt1/detection_loss‚2.048320030172666 /loss/FBt3/detection_loss‚0.7085822801394709 _runtime‚13608.773348807 2loss/FBt4/assignment_loss‚0.018984048161605215 %loss/total_loss‚39.81766299366326 0loss/FRt1/assignment_loss‚2.4738003669715507 1loss/FBt3/assignment_loss‚0.02540660327971647 0loss/SRqqt2/detection_loss‚2.205813929438591 1loss/SRqqt4/detection_loss‚2.7370397514767117 .loss/FBt1/detection_loss‚1.903467671945691 .loss/FBt4/detection_loss‚3.045921194553375 1loss/FBt1/assignment_loss‚0.08188762918139147 /loss/FRt2/detection_loss‚2.1864586199773477 1loss/SRqqt3/detection_loss‚0.7618946262469916 /loss/FRt4/detection_loss‚3.3004984829736794²áP  ’¤.‚(:‡ þ€èÔðÒ·† gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚117.193' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.29825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0oìçL}¥.‚(:ò þ€èÔ¡ã­‡ disk.sda1.out‚0 disk.sdb2.in‚5.796875 disk.rbd2.out‚ 3.87109375# proc.memory.rssMB‚ 3032.23828125 proc.cpu.threads‚17! disk.dm-0.out‚160278.02734375 disk.rbd0.out‚ 3.646484375 disk.sdb.out‚1516.7431640625% memory_percent‚41.394591331481934) proc.memory.percent‚18.50731372833252! disk.sda2.out‚1516.7431640625 network.sent‚ 3.0972264e+07$ disk./.usageGB‚2973.761016845703 disk.md0p1.in‚ 11.953125 disk.md0p1.out‚ 1461.7734375! disk.sdb2.out‚1516.7431640625 disk.rbd0.in‚0.015625 disk.sda.out‚1516.7431640625 disk.sdb1.out‚0) disk./.usagePercent‚83.14760264668297" disk.nvme0n1.out‚ 83655.0546875 disk.md0.out‚ 1461.7734375# disk.nvme1n1.in‚201585.86328125 disk.sdb.in‚5.796875 disk.sda.in‚6.15625 disk.rbd2.in‚0 disk.sdb1.in‚0 cpu‚25.1998945005188 network.recv‚ 4.680228e+06" disk.nvme0n1.in‚330194.6484375 disk.sda1.in‚0# disk.nvme1n1.out‚76622.97265625 disk.dm-0.in‚531780.51171875 disk.sda2.in‚6.15625 disk.md0.in‚ 11.953125) proc.memory.availableMB‚ 9601.91015625 óu‘¦.‚(:† èÔ´é… gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚140.478' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.1195 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>º0L}§.‚(:ò èÔÇ disk.rbd2.out‚ 3.87109375# disk.nvme1n1.in‚201585.86328125 disk.sda2.out‚1518.888671875 disk.sdb2.out‚1518.888671875 disk.md0.out‚ 1463.85546875 disk.rbd2.in‚0 disk.sdb2.in‚5.796875 disk.md0.in‚ 11.953125 disk.sdb.in‚5.796875 cpu‚25.199712967592248% disk./.usageGB‚2973.7839584350586 disk.rbd0.out‚ 3.646484375" disk.nvme0n1.in‚330194.6484375# disk.nvme0n1.out‚83724.31640625 disk.dm-0.in‚531780.51171875 disk.dm-0.out‚ 160347.453125 disk.sda1.in‚0 network.sent‚ 3.1079991e+07 disk.rbd0.in‚0.015625 disk.sdb1.in‚0 disk.sda1.out‚0 disk.md0p1.out‚ 1463.85546875% memory_percent‚41.394639015197754# disk.nvme1n1.out‚76623.13671875 disk.sda.in‚6.15625# proc.memory.rssMB‚ 3032.23828125 network.recv‚ 4.687472e+06 disk.sdb.out‚1518.888671875 disk.sda.out‚1518.888671875 disk.sda2.in‚6.15625) proc.memory.percent‚18.50731372833252 proc.cpu.threads‚17 disk.md0p1.in‚ 11.953125 disk.sdb1.out‚0) proc.memory.availableMB‚ 9601.90234375) disk./.usagePercent‚83.14824410312316'&u\’¨.‚(:‡ œèÔÆŠ‡† gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚134.575' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.64375 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f/Ô}©.‚(:ò œèÔÖˆ‡ disk.sda.out‚1520.3115234375 disk.sda1.out‚0) proc.memory.availableMB‚ 9602.12890625 disk.dm-0.in‚531780.51171875 disk.md0p1.out‚ 1465.21484375 disk.md0.in‚ 11.953125 disk.sdb.out‚1520.3115234375 disk.sdb1.out‚0 disk.md0p1.in‚ 11.953125) proc.memory.percent‚18.50731372833252 proc.cpu.threads‚17 disk.sda2.in‚6.15625% disk./.usageGB‚2973.6934814453125 disk.sdb2.in‚5.796875 disk.rbd0.in‚0.015625 cpu‚25.199530414849537 network.sent‚ 3.108503e+07 disk.rbd0.out‚ 3.646484375# disk.nvme1n1.in‚201585.86328125 disk.sdb1.in‚0 disk.dm-0.out‚ 160376.59375! disk.sda2.out‚1520.3115234375% memory_percent‚41.393256187438965# proc.memory.rssMB‚ 3032.23828125! disk.sdb2.out‚1520.3115234375 disk.rbd2.in‚0 disk.sda1.in‚0" disk.nvme1n1.out‚ 76623.2734375 network.recv‚ 4.691082e+06 disk.md0.out‚ 1465.21484375 disk.sda.in‚6.15625" disk.nvme0n1.in‚330194.6484375" disk.nvme0n1.out‚ 83753.3203125) disk./.usagePercent‚83.14571432862229 disk.rbd2.out‚ 3.87109375 disk.sdb.in‚5.796875Â=±"4ª./ _wandbruntime‚13660  _runtime‚13660WU¯BÉ«.à & lr-AdamW/pg2‚5.018561283162145e-06 trainer/global_step‚11549 % _timestamp‚1.7884776076347537e+09 _runtime‚13660.819676937 _step‚947 & lr-AdamW/pg1‚5.018561283162145e-06 †1‘ ¬.È àQŽáÄ_step‚947 & lr-AdamW/pg1‚5.018561283162145e-06 & lr-AdamW/pg2‚5.018561283162145e-06 trainer/global_step‚11549 % _timestamp‚1.7884776076347537e+09 _runtime‚13660.819676937³øíš94­./ _wandbruntime‚13660  _runtime‚13660È'Ôò'®.ìO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 _step‚948 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚470818716360704 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22955821118464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22890512567040 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚470872297117868 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 533959676 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 314149341 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚11549 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 219810335 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚493774285807616 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 314122997 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884776076358445e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 214870202 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 219810335 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚704 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 533933330 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 314149341 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚470818698357760 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 131395514 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚83474690 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23342917154816 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚493774537479168 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚498682091746816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚13660.820873282 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚470818716360704 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚498682067806720 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23342628632576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚470818698357760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚447981555569744 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 219810333 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22955587449856 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚522024696439296 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22890741548124 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚493774285807616 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 219810333 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚493774537479168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚83475392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 314122997 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚522025008901632¸À%÷'¯.ñO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 314122997 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚522025008901632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 _step‚948 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚470818716360704 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22955821118464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22890512567040 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚470872297117868 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 533959676 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 314149341 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚11549 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 219810335 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚493774285807616 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 314122997 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884776076358445e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 214870202 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 219810335 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚704 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 533933330 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 314149341 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚470818698357760 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 131395514 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚83474690 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23342917154816 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚493774537479168 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚498682091746816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚13660.820873282 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚470818716360704 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚498682067806720 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23342628632576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚470818698357760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚447981555569744 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 219810333 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22955587449856 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚522024696439296 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22890741548124 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚493774285807616 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 219810333 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚493774537479168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚83475392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608´‹íÖw4°./ _wandbruntime‚13661  _runtime‚13661ãŽû&±.õM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚83482505 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚470857984394752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23344626011136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 533978297 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 % _timestamp‚1.7884776085866337e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 219828608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚498723775716864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 trainer/global_step‚11549 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 219828608 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚470911539943834 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚448019085927168 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚470911785964486 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚470857984394752 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 314149689 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚448019068887808 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 533978297 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 534004669 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 214888821 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚522068688139776 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22957551475712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22957785157632 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 131406316 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 _runtime‚13661.77227567 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22957551475712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚652 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22892471056026 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚470858002397696 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 214888169 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22892700037318 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚949 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 314176059 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370IJ ¿².úM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚83482505 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚470857984394752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23344626011136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 533978297 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 % _timestamp‚1.7884776085866337e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 219828608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚498723775716864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 trainer/global_step‚11549 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.pe[ÚðAak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 219828608 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚470911539943834 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚448019085927168 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚470911785964486 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚470857984394752 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 314149689 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚448019068887808 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 533978297 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 534004669 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 214888821 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚522068688139776 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22957551475712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22957785157632 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 131406316 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 _runtime‚13661.77227567 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22957551475712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚652 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22892471056026 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚470858002397696 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 214888169 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22892700037318 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚949 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 314176059 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0µÿäò4³./ _wandbruntime‚13661  _runtime‚13661yŸ³Z´.Ô 0loss/FRt3/assignment_loss‚0.9701588942724115 .loss/FRt2/detection_loss‚1.796797530839897 .loss/FRt4/detection_loss‚3.259678224270994 1loss/SRqqt4/detection_loss‚2.1417267467731085 /loss/FRt2/assignment_loss‚3.379819299957969 1loss/SRqqt1/assignment_loss‚1.893571657261678 2loss/SRqqt2/assignment_loss‚1.3308058742965971 1loss/SRqqt1/detection_loss‚2.5054620632103513 %loss/total_loss‚41.57573428733622 % _timestamp‚1.7884776085885959e+09 1loss/FBt4/assignment_loss‚0.39044217917609736 /loss/FRt1/detection_loss‚2.6406166553497314 0loss/SRqqt2/detection_loss‚2.321709818499429 .loss/FBt4/detection_loss‚2.607174429628584 trainer/global_step‚11549 _runtime‚13661.773332825 0loss/FBt2/assignment_loss‚0.0679190206234684 -loss/FBt1/detection_loss‚2.31863972875807 /loss/FBt2/detection_loss‚2.4935039480527244 /loss/FBt3/detection_loss‚0.5705534340048285 epoch‚9 0loss/FRt4/assignment_loss‚3.2010579163377937 2loss/SRqqt3/assignment_loss‚0.8706383126787841 1loss/FBt1/assignment_loss‚0.04193922702991889 1loss/SRqqt3/detection_loss‚0.7656231450211061 1loss/FBt3/assignment_loss‚0.06630558647518835 _step‚950 /loss/FRt1/assignment_loss‚3.044426890519949 2loss/SRqqt4/assignment_loss‚1.5457334933826365 .loss/FRt3/detection_loss‚1.351430210914901+ô8Ÿ_µ.Ù 0loss/SRqqt2/detection_loss‚2.321709818499429 .loss/FBt4/detection_loss‚2.607174429628584 trainer/global_step‚11549 _runtime‚13661.773332825 0loss/FBt2/assignment_loss‚0.0679190206234684 -loss/FBt1/detection_loss‚2.31863972875807 /loss/FBt2/detection_loss‚2.4935039480527244 /loss/FBt3/detection_loss‚0.5705534340048285 epoch‚9 0loss/FRt4/assignment_loss‚3.2010579163377937 2loss/SRqqt3/assignment_loss‚0.8706383126787841 1loss/FBt1/assignment_loss‚0.04193922702991889 1loss/SRqqt3/detection_loss‚0.7656231450211061 1loss/FBt3/assignment_loss‚0.06630558647518835 _step‚950 /loss/FRt1/assignment_loss‚3.044426890519949 2loss/SRqqt4/assignment_loss‚1.5457334933826365 .loss/FRt3/detection_loss‚1.351430210914901 0loss/FRt3/assignment_loss‚0.9701588942724115 .loss/FRt2/detection_loss‚1.796797530839897 .loss/FRt4/detection_loss‚3.259678224270994 1loss/SRqqt4/detection_loss‚2.1417267467731085 /loss/FRt2/assignment_loss‚3.379819299957969 1loss/SRqqt1/assignment_loss‚1.893571657261678 2loss/SRqqt2/assignment_loss‚1.3308058742965971 1loss/SRqqt1/detection_loss‚2.5054620632103513 %loss/total_loss‚41.57573428733622 % _timestamp‚1.7884776085885959e+09 1loss/FBt4/assignment_loss‚0.39044217917609736 /loss/FRt1/detection_loss‚2.6406166553497314¶‡ï¹í¶.‚(:… «èÔÂô͆ gpu.0.gpu‚42.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚143.872' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.968 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÈéWØ…·.‚(:ú «èÔ–Ãü‡ network.sent‚ 3.1103242e+07 disk.rbd0.in‚0.015625# disk.nvme1n1.in‚201585.86328125 disk.sdb1.out‚0! disk.dm-0.out‚160478.90234375 disk.md0p1.in‚ 11.953125# proc.memory.rssMB‚ 3032.23828125) disk./.usagePercent‚83.15180528496964 disk.md0.in‚ 11.953125 disk.md0.out‚ 1466.90234375 cpu‚25.19932666419547 network.recv‚ 4.694928e+06 disk.rbd0.out‚ 3.646484375 disk.sdb.in‚5.796875! disk.sdb.out‚1522.08251953125" disk.nvme0n1.in‚330194.6484375 disk.rbd2.out‚ 3.87109375# disk.nvme1n1.out‚76623.66015625) proc.memory.percent‚18.50731372833252 disk.sda.in‚6.15625 disk.sda2.in‚6.15625 disk.md0p1.out‚ 1466.90234375 disk.sdb2.in‚5.796875) proc.memory.availableMB‚ 9601.88671875 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sdb1.in‚0! disk.sda.out‚1522.08251953125 disk.sda1.out‚0" disk.nvme0n1.out‚ 83855.2421875" disk.sdb2.out‚1522.08251953125% disk./.usageGB‚2973.9113235473633 disk.dm-0.in‚531780.51171875 disk.sda1.in‚0" disk.sda2.out‚1522.08251953125% memory_percent‚41.394734382629395½8±’¸.‚(:‡ ºèÔÿ¾‚† gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚116.583' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.14575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âY#N¹.‚(:ô ºèÔ›©¯‡! disk.sda.out‚1524.38037109375 disk.sda1.in‚0 disk.md0p1.out‚ 1469.1171875 disk.rbd0.in‚0.015625) proc.memory.percent‚18.50731372833252 disk.sda1.out‚0% disk./.usageGB‚2974.0921745300293 disk.dm-0.in‚531780.51171875! disk.dm-0.out‚160659.83984375 disk.md0p1.in‚ 11.953125 disk.sdb2.in‚5.796875 proc.cpu.threads‚17 network.sent‚ 3.119848e+07 disk.sdb1.in‚0" disk.nvme0n1.out‚ 83943.3359375" disk.sdb2.out‚1524.38037109375 disk.md0.in‚ 11.953125 disk.md0.out‚ 1469.1171875 disk.rbd0.out‚ 3.646484375# disk.nvme1n1.out‚76716.50390625 disk.sdb1.out‚0 disk.sdb.in‚5.796875( proc.memory.availableMB‚ 9601.8203125" disk.sda2.out‚1524.38037109375 disk.rbd2.in‚0$ memory_percent‚41.39513969421387 disk.rbd2.out‚ 3.87109375# disk.nvme1n1.in‚201585.86328125# proc.memory.rssMB‚ 3032.23828125! disk.sdb.out‚1524.38037109375 cpu‚25.199110999341066 network.recv‚ 4.702182e+06' disk./.usagePercent‚83.156861954137 disk.sda.in‚6.15625" disk.nvme0n1.in‚330194.6484375 disk.sda2.in‚6.15625U‘º.‚(:† ÉèÔ朣‡ gpu.0.gpu‚6.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚112.849' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.21225 gpu.0.smClock‚1185 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A…9‚».‚(:÷ ÉèÔöÙšˆ' proc.memory.availableMB‚ 9602.078125 cpu‚25.198908406442477! disk.dm-0.out‚160840.00390625 disk.sda1.out‚0" disk.nvme0n1.in‚330194.6484375 disk.sda2.in‚6.15625 network.sent‚ 3.1204307e+07% disk./.usageGB‚2974.0910034179688 disk.md0.out‚ 1470.62890625 disk.sda.in‚6.15625# disk.nvme0n1.out‚84030.69921875 disk.sda1.in‚0 disk.md0p1.in‚ 11.953125) disk./.usagePercent‚83.15682920935363# disk.nvme1n1.in‚201585.86328125 disk.sdb1.out‚0# proc.memory.rssMB‚ 3032.23828125 proc.cpu.threads‚17" disk.sdb2.out‚1525.95263671875 disk.md0.in‚ 11.953125 disk.rbd0.out‚ 3.646484375 disk.rbd2.out‚ 3.87109375! disk.sdb.out‚1525.95263671875 disk.sdb1.in‚0 disk.dm-0.in‚531780.51171875 disk.rbd0.in‚0.015625 disk.rbd2.in‚0" disk.nvme1n1.out‚ 76809.3046875" disk.sda2.out‚1525.95263671875) proc.memory.percent‚18.50731372833252 disk.sdb2.in‚5.796875! disk.sda.out‚1525.95263671875 network.recv‚ 4.706536e+06 disk.md0p1.out‚ 1470.62890625 disk.sdb.in‚5.796875# memory_percent‚41.3935661315918PXÛs’¼.‚(:‡ ØèÔ“®Ÿ† gpu.0.gpu‚12.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚143.595' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.89875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ál °{½.‚(:ð ØèÔ¸“‡‡ network.recv‚ 4.710787e+06 disk.rbd0.out‚ 3.646484375 disk.rbd2.in‚0 disk.sda1.out‚0 disk.md0p1.out‚ 1472.203125! disk.dm-0.out‚160944.69921875! disk.sda2.out‚1527.6064453125 disk.md0p1.in‚ 11.953125) proc.memory.availableMB‚ 9602.06640625) disk./.usagePercent‚83.14931358827874" disk.nvme0n1.in‚330194.6484375 disk.sda2.in‚6.15625! disk.sdb2.out‚1527.6064453125$ memory_percent‚41.39363765716553# proc.memory.rssMB‚ 3032.23828125 cpu‚25.19869571515517 disk.sdb1.in‚0 disk.rbd2.out‚ 3.87109375" disk.nvme1n1.out‚ 76809.4765625 disk.md0.in‚ 11.953125# disk.nvme1n1.in‚201585.86328125 disk.sdb.in‚5.796875 disk.dm-0.in‚531780.51171875 disk.sda1.in‚0 disk.md0.out‚ 1472.203125 network.sent‚ 3.1210332e+07 disk.sdb.out‚1527.6064453125 disk.sda.in‚6.15625 disk.sda.out‚1527.6064453125$ disk./.usageGB‚2973.822208404541 disk.rbd0.in‚0.015625 disk.sdb1.out‚0# disk.nvme0n1.out‚84135.22265625 disk.sdb2.in‚5.796875) proc.memory.percent‚18.50731372833252 proc.cpu.threads‚17¶¿4¾./ _wandbruntime‚13712  _runtime‚13712ÂXåÉ¿.à & lr-AdamW/pg1‚4.135265681027407e-06 & lr-AdamW/pg2‚4.135265681027407e-06 trainer/global_step‚11599 % _timestamp‚1.7884776595786169e+09 _runtime‚13712.763649109 _step‚951%cßvÎÀ.È & lr-AdamW/pg2‚4.135265681027407e-06 trainer/global_step‚11599 % _timestamp‚1.7884776595786169e+09 _runtime‚13712.763649109 _step‚951 & lr-AdamW/pg1‚4.135265681027407e-06·Äà†4Á./ _wandbruntime‚13712  _runtime‚13712Òù×ò'Â.ìO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 131935614 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23054036825088 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23054036825088 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 536211378 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23442730131456 % _timestamp‚1.7884776595797884e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 220724083 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23442435317760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚500766266309120 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚449858056401584 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 315460949 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _runtime‚13712.764924525 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚524208701626880 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚472846753029576 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚472846507009132 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚472783000207360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 315487293 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚449858073440944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚495836803363840 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚524209020380672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚495837055035392 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 215754852 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚500766290249216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚83819930 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294813696 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚11599 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 131935612 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 536185032 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 315487293 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 220724083 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚472783018210304 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚495836803363840 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 _step‚952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚472783000207360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23053803156480 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚692 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚495837055035392 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 215755544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22988679588632 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 315460949 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23053803156480 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 220724085 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 220724085 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚472783018210304 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚83819240 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346Ò1“'Ã.ñO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚472783018210304 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚495836803363840 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 _step‚952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚472783000207360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23053803156480 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚692 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚495837055035392 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 215755544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22988679588632 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 315460949 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23053803156480 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 220724085 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 220724085 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚472783018210304 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚83819240 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 131935614 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23054036825088 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23054036825088 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 536211378 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23442730131456 % _timestamp‚1.7884776595797884e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 220724083 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23442435317760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚500766266309120 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚449858056401584 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 315460949 =6DeviéÀa±ÚceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _runtime‚13712.764924525 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚524208701626880 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚472846753029576 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚472846507009132 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚472783000207360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 315487293 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚449858073440944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚495836803363840 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚524209020380672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚495837055035392 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 215754852 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚500766290249216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚83819930 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294813696 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚11599 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 131935612 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 536185032 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 315487293 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 220724083 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2¸‘bà:4Ä./ _wandbruntime‚13713  _runtime‚13713½rû&Å.õM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 215772305 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 220742358 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚472822304247296 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚524252696079872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 536229999 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚495878053426688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22990409096534 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 % _timestamp‚1.7884776607094269e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22990638077826 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 220742360 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚449895603798368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚500807974219264 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚472822286244352 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚472885995855542 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚23055767182336 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23055767182336 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚524252379436544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚694 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 131946416 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23056000864256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 trainer/global_step‚11599 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚472822304247296 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23056000864256 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 220742358 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚472886241876194 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚495878305111552 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 315514011 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 536256371 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 315514011 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 536256371 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23444429157376 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚500807950279168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚13713.89600344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚83825891 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104C1"'Æ.úM RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 536256371 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 315514011 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 536256371 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23444429157376 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚500807950279168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚13713.89600344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚83825891 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 215772305 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 220742358 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚472822304247296 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚524252696079872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 536229999 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚495878053426688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22990409096534 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 % _timestamp‚1.7884776607094269e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22990638077826 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 220742360 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚449895603798368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚500807974219264 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚472822286244352 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚472885995855542 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚23055767182336 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23055767182336 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚524252379436544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚694 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 131946416 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23056000864256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 trainer/global_step‚11599 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚472822304247296 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23056000864256 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 220742358 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚472886241876194 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚495878305111552 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 315514011 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0¹epÒ¿4Ç./ _wandbruntime‚13713  _runtime‚13713ÿGhcÈ.Ý 2loss/SRqqt2/assignment_loss‚1.2679747265318166 0loss/FBt2/assignment_loss‚0.1109759472084097 1loss/SRqqt1/detection_loss‚1.8221859031397363 1loss/SRqqt3/detection_loss‚0.8324304399129592 1loss/FBt4/assignment_loss‚0.17121545130458152 /loss/FRt2/detection_loss‚2.2188731044014096 trainer/global_step‚11599 % _timestamp‚1.7884776607124968e+09 _step‚954 .loss/FBt2/detection_loss‚2.193203114800983 _runtime‚13713.897286906 0loss/FRt2/assignment_loss‚2.7206279271986427 0loss/FRt4/assignment_loss‚2.1632935350591485 2loss/SRqqt3/assignment_loss‚0.9329004494963508 2loss/SRqqt4/assignment_loss‚1.2606945210695266 1loss/FBt1/assignment_loss‚0.11741395912819712 0loss/FBt3/detection_loss‚0.31773024343873385 epoch‚9 2loss/SRqqt1/assignment_loss‚1.2486782746146554 &loss/total_loss‚39.008253339141994 0loss/FRt3/assignment_loss‚1.2759113028401234 1loss/FBt3/assignment_loss‚0.08806702934417923 .loss/FRt1/detection_loss‚2.605821568518877 .loss/FRt3/detection_loss‚1.320995012603578 /loss/FRt4/detection_loss‚3.0205099311741916 1loss/SRqqt2/detection_loss‚2.7838668434516243 1loss/SRqqt4/detection_loss‚2.5281651401519776 /loss/FBt4/detection_loss‚2.3018406744186697 /loss/FBt1/detection_loss‚2.7751257181167603 0loss/FRt1/assignment_loss‚2.9297525212168694²ÝÚehÉ.â 0loss/FRt3/assignment_loss‚1.2759113028401234 1loss/FBt3/assignment_loss‚0.08806702934417923 .loss/FRt1/detection_loss‚2.605821568518877 .loss/FRt3/detection_loss‚1.320995012603578 /loss/FRt4/detection_loss‚3.0205099311741916 1loss/SRqqt2/detection_loss‚2.7838668434516243 1loss/SRqqt4/detection_loss‚2.5281651401519776 /loss/FBt4/detection_loss‚2.3018406744186697 /loss/FBt1/detection_loss‚2.7751257181167603 0loss/FRt1/assignment_loss‚2.9297525212168694 2loss/SRqqt2/assignment_loss‚1.2679747265318166 0loss/FBt2/assignment_loss‚0.1109759472084097 1loss/SRqqt1/detection_loss‚1.8221859031397363 1loss/SRqqt3/detection_loss‚0.8324304399129592 1loss/FBt4/assignment_loss‚0.17121545130458152 /loss/FRt2/detection_loss‚2.2188731044014096 trainer/global_step‚11599 % _timestamp‚1.7884776607124968e+09 _step‚954 .loss/FBt2/detection_loss‚2.193203114800983 _runtime‚13713.897286906 0loss/FRt2/assignment_loss‚2.7206279271986427 0loss/FRt4/assignment_loss‚2.1632935350591485 2loss/SRqqt3/assignment_loss‚0.9329004494963508 2loss/SRqqt4/assignment_loss‚1.2606945210695266 1loss/FBt1/assignment_loss‚0.11741395912819712 0loss/FBt3/detection_loss‚0.31773024343873385 epoch‚9 2loss/SRqqt1/assignment_loss‚1.2486782746146554 &loss/total_loss‚39.008253339141994ºM{½æ’Ê.‚(:‡ çèԸŖ‡ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚136.743' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.18575 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êÏq~Ë.‚(:ó çèÔªŸþ‡ disk.sda.in‚6.15625 disk.sda2.in‚6.15625 proc.cpu.threads‚17 network.sent‚ 3.1215589e+07 network.recv‚ 4.714496e+06) disk./.usagePercent‚83.14933225387186 disk.md0.in‚ 11.953125 disk.md0.out‚ 1474.140625 disk.rbd2.in‚0" disk.nvme0n1.in‚330194.6484375# proc.memory.rssMB‚ 3032.23828125 disk.rbd2.out‚ 3.87109375 disk.sdb.in‚5.796875# disk.nvme0n1.out‚84178.15234375% memory_percent‚41.393160820007324 cpu‚25.19849373266576% disk./.usageGB‚2973.8228759765625 disk.sdb2.in‚5.796875# disk.nvme1n1.in‚201585.86328125! disk.sdb.out‚1529.59423828125 disk.dm-0.out‚ 160993.8125 disk.rbd0.in‚0.015625 disk.sda1.out‚0) proc.memory.percent‚18.50731372833252 disk.rbd0.out‚ 3.646484375# disk.nvme1n1.out‚76815.66015625 disk.dm-0.in‚531780.51171875" disk.sda2.out‚1529.59423828125 disk.md0p1.in‚ 11.953125 disk.sdb1.in‚0 disk.sda1.in‚0 disk.md0p1.out‚ 1474.140625) proc.memory.availableMB‚ 9602.14453125" disk.sdb2.out‚1529.59423828125 disk.sdb1.out‚0! disk.sda.out‚1529.59423828125ï1œ‘Ì.‚(:† öèÔò€æ‡ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚136.134' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.0335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿}äwÍ.‚(:ì öèÔÝï׈ disk.sdb2.in‚5.796875$ disk./.usageGB‚2974.096347808838 disk.md0.out‚ 1475.7109375 disk.rbd2.out‚ 3.87109375 disk.sdb1.out‚0" disk.nvme0n1.in‚330194.6484375 disk.dm-0.in‚531780.51171875 disk.md0p1.in‚ 11.953125 cpu‚25.198293024115543 disk.sda.in‚6.15625# disk.nvme0n1.out‚84178.71484375 disk.sda1.out‚0& proc.memory.availableMB‚ 9602.03125# disk.nvme1n1.in‚201585.86328125 network.sent‚ 3.1323745e+07 disk.rbd0.in‚0.015625 disk.sda2.in‚6.15625 disk.md0p1.out‚ 1475.7109375 network.recv‚ 4.721998e+06) proc.memory.percent‚18.50731372833252 proc.cpu.threads‚17! disk.dm-0.out‚161135.05078125 disk.rbd2.in‚0 disk.sdb.in‚5.796875# proc.memory.rssMB‚ 3032.23828125 disk.rbd0.out‚ 3.646484375" disk.nvme1n1.out‚ 76956.3359375 disk.sdb1.in‚0 disk.sda.out‚1531.228515625 disk.sda2.out‚1531.228515625 disk.sdb2.out‚1531.228515625 disk.md0.in‚ 11.953125 disk.sdb.out‚1531.228515625 disk.sda1.in‚0$ memory_percent‚41.39385223388672) disk./.usagePercent‚83.15697864075914f€:=yÎ.‚(:î …‚èÔïòÛ‡ disk.sdb.in‚5.796875 disk.dm-0.in‚531780.51171875 disk.sda1.out‚0 disk.sda2.in‚6.15625! disk.sda2.out‚1532.7939453125# proc.memory.rssMB‚ 3032.23828125 disk.sdb.out‚1532.7939453125 disk.rbd0.out‚ 3.646484375 disk.sda.out‚1532.7939453125" disk.nvme0n1.in‚330194.6484375! disk.sdb2.out‚1532.7939453125 disk.rbd2.in‚0$ memory_percent‚41.39375686645508! disk.dm-0.out‚161281.51953125 network.sent‚ 3.1329461e+07 network.recv‚ 4.72604e+06% disk./.usageGB‚2974.1291847229004 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sdb2.in‚5.796875' proc.memory.availableMB‚ 9602.046875) disk./.usagePercent‚83.15789677461964 disk.sda1.in‚0 disk.md0p1.in‚ 11.953125 disk.md0.out‚ 1477.21484375# disk.nvme1n1.in‚201585.86328125# disk.nvme1n1.out‚77102.64453125 cpu‚25.19811725714185 proc.cpu.threads‚17 disk.md0.in‚ 11.953125) proc.memory.percent‚18.50731372833252 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 3.87109375 disk.sda.in‚6.15625 disk.nvme0n1.out‚ 84178.875 disk.md0p1.out‚ 1477.214843750TZÙ Ï.‚(:‡ …‚èÔŸö⇠gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚144.821' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚3TÐg†36.20525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>[«R4Ð./ _wandbruntime‚13768  _runtime‚13768^ ºþËÑ.Å trainer/global_step‚11649 % _timestamp‚1.7884777152705214e+09 _runtime‚13768.455613612 _step‚955 ' lr-AdamW/pg1‚3.3370156494266347e-06 ' lr-AdamW/pg2‚3.3370156494266347e-06Fê(ÿÐÒ.Ê _step‚955 ' lr-AdamW/pg1‚3.3370156494266347e-06 ' lr-AdamW/pg2‚3.3370156494266347e-06 trainer/global_step‚11649 % _timestamp‚1.7884777152705214e+09 _runtime‚13768.455613612»ÊI™×4Ó./ _wandbruntime‚13768  _runtime‚13768,åÎò'Ô.ìO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚497899320920064 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 216642855 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 trainer/global_step‚11649 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 316798901 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚451734591312144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 132475712 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚526392621073408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 221637833 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23152252531712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 322948096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 538463080 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚13768.456714077 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23152018863104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23152252531712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚84167141 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 % _timestamp‚1.7884777152713788e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 538436734 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23152018863104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚451734574272784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 316825245 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23086617629140 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚474820962920840 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚84166467 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 221637835 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚526392944021504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚956 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 316825245 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚474747320059904 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 216642179 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 132475714 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136~x…÷'Õ.ñO N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 216642179 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 132475714 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚497899320920064 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 216642855 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 trainer/global_step‚11649 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 316798901 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚451734591312144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 132475712 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚526392621073408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 221637833 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23152252531712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 322948096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 538463080 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚13768.456714077 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23152018863104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23152252531712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚84167141 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 % _timestamp‚1.7884777152713788e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 538436734 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23152018863104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚451734574272784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 316825245 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23086617629140 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚474820962920840 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚84166467 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 221637835 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚526392944021504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚956 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 316825245 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚474747320059904¼ŸËck4Ö./ _wandbruntime‚13769  _runtime‚13769˜?Ððü&×.öM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚497940570982912 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23544161761280 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚474786606096896 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚23153982888960 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23154216570880 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 132486516 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚502892148781568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚451772121669568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚633 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 316851963 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚526436310542848 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚497940822667776 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 538508073 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23544458658816 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚23088576118334 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚84173468 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚474786588093952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 216659982 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23153982888960 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 221656108 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚474786606096896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 _runtime‚13769.580584669 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚474860451767250 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚474860697787902 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 538481701 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 316851963 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚497940822667776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0sÅTØ.ûM XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚451772121669568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚633 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 316851963 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚526436310542848 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚49794!ué 0822667776 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 538508073 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23544458658816 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚23088576118334 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚84173468 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚474786588093952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 216659982 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23153982888960 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 221656108 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚474786606096896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 _runtime‚13769.580584669 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚474860451767250 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚474860697787902 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 538481701 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 316851963 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚497940822667776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚497940570982912 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23544161761280 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚474786606096896 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚23153982888960 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23154216570880 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 132486516 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚502892148781568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712½x˜ —4Ù./ _wandbruntime‚13769  _runtime‚13769õ‘ ˆ_Ú.Ù .loss/FBt2/detection_loss‚3.149565353989601 /loss/FBt3/detection_loss‚0.4297548085295906 _step‚958 /loss/FRt2/assignment_loss‚3.474696390482844 1loss/SRqqt3/detection_loss‚0.8102705723521384 /loss/FRt4/assignment_loss‚3.484718307852745 /loss/FRt3/detection_loss‚1.4388027729518298 _runtime‚13769.582746539 0loss/FRt3/assignment_loss‚1.3614101186394691 .loss/FRt4/detection_loss‚3.156951746592919 0loss/SRqqt1/detection_loss‚1.958492949604988 epoch‚9 0loss/FRt1/assignment_loss‚2.1584199306211973 2loss/SRqqt2/assignment_loss‚1.4859870767507415 1loss/FBt3/assignment_loss‚0.06974438947318783 /loss/FBt4/detection_loss‚2.7294850604874745 2loss/SRqqt3/assignment_loss‚1.1082215664950623 1loss/FBt1/assignment_loss‚0.07890198396376036 /loss/FRt2/detection_loss‚1.8076156517063096 1loss/SRqqt2/detection_loss‚2.5791960725417504 /loss/FBt1/detection_loss‚2.5763106431279863 %loss/total_loss‚42.21544910041957 2loss/SRqqt1/assignment_loss‚1.4903617877513171 2loss/SRqqt4/assignment_loss‚1.1759613159423072 .loss/FRt1/detection_loss‚2.477939917853004 trainer/global_step‚11649 1loss/FBt4/assignment_loss‚0.27019855672759263 % _timestamp‚1.7884777163978667e+09 1loss/FBt2/assignment_loss‚0.19128960311979792 1loss/SRqqt4/detection_loss‚2.7511525228619576¨ždÛ.Þ % _timestamp‚1.7884777163978667e+09 1loss/FBt2/assignment_loss‚0.19128960311979792 1loss/SRqqt4/detection_loss‚2.7511525228619576 .loss/FBt2/detection_loss‚3.149565353989601 /loss/FBt3/detection_loss‚0.4297548085295906 _step‚958 /loss/FRt2/assignment_loss‚3.474696390482844 1loss/SRqqt3/detection_loss‚0.8102705723521384 /loss/FRt4/assignment_loss‚3.484718307852745 /loss/FRt3/detection_loss‚1.4388027729518298 _runtime‚13769.582746539 0loss/FRt3/assignment_loss‚1.3614101186394691 .loss/FRt4/detection_loss‚3.156951746592919 0loss/SRqqt1/detection_loss‚1.958492949604988 epoch‚9 0loss/FRt1/assignment_loss‚2.1584199306211973 2loss/SRqqt2/assignment_loss‚1.4859870767507415 1loss/FBt3/assignment_loss‚0.06974438947318783 /loss/FBt4/detection_loss‚2.7294850604874745 2loss/SRqqt3/assignment_loss‚1.1082215664950623 1loss/FBt1/assignment_loss‚0.07890198396376036 /loss/FRt2/detection_loss‚1.8076156517063096 1loss/SRqqt2/detection_loss‚2.5791960725417504 /loss/FBt1/detection_loss‚2.5763106431279863 %loss/total_loss‚42.21544910041957 2loss/SRqqt1/assignment_loss‚1.4903617877513171 2loss/SRqqt4/assignment_loss‚1.1759613159423072 .loss/FRt1/detection_loss‚2.477939917853004 trainer/global_step‚11649 1loss/FBt4/assignment_loss‚0.27019855672759263¾/P}¾Ü.‚(:ö ”‚èÔ¯÷Έ disk.rbd0.in‚0.015625 disk.dm-0.out‚ 161492.671875 disk.md0p1.in‚ 11.953125# proc.memory.rssMB‚ 3032.23828125 network.recv‚ 4.7305e+06" disk.sdb2.out‚1534.40380859375 disk.md0.out‚ 1478.76171875# disk.nvme0n1.out‚84178.94921875 disk.sda2.in‚6.15625 disk.sdb1.in‚0" disk.nvme0n1.in‚330194.6484375 disk.sda1.out‚0 disk.md0p1.out‚ 1478.76171875) proc.memory.availableMB‚ 9602.22265625 disk.md0.in‚ 11.953125 disk.sdb1.out‚0 disk.sda.in‚6.15625" disk.sda2.out‚1534.40380859375$ memory_percent‚41.39268398284912) disk./.usagePercent‚83.17099927436706% disk./.usageGB‚2974.5977935791016 disk.sdb2.in‚5.796875 disk.rbd2.out‚ 3.87109375# disk.nvme1n1.in‚201585.86328125 disk.sdb.in‚5.796875! disk.sda.out‚1534.40380859375 disk.sda1.in‚0 proc.cpu.threads‚17 disk.rbd2.in‚0# disk.nvme1n1.out‚77313.72265625 disk.dm-0.in‚531780.51171875) proc.memory.percent‚18.50731372833252 cpu‚25.197934753801867 network.sent‚ 3.134856e+07 disk.rbd0.out‚ 3.646484375! disk.sdb.out‚1534.40380859375ZH"‘Ý.‚(:† ”‚èÔĬ‰ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚138.642' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.6605 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p>c‡’Þ.‚(:‡ £‚èÔø°å… gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚142.653' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.66325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šk©,~ß.‚(:ó £‚èÔ•¡ä†! disk.sdb.out‚1536.38623046875 disk.sda1.in‚0 cpu‚25.197781157425265! disk.sda.out‚1536.38623046875 proc.cpu.threads‚17% disk./.usageGB‚2974.5978088378906 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sdb2.in‚5.796875 disk.md0.in‚ 11.953125) proc.memory.percent‚18.50731372833252 network.recv‚ 4.737144e+06 disk.rbd0.in‚0.015625" disk.nvme0n1.in‚330194.6484375 disk.dm-0.in‚531780.51171875" disk.sdb2.out‚1536.38623046875$ memory_percent‚41.39575958251953# proc.memory.rssMB‚ 3032.23828125# disk.nvme1n1.in‚201585.86328125! disk.nvme1n1.out‚ 77573.171875 disk.sda.in‚6.15625 disk.dm-0.out‚161752.3203125 disk.sda2.in‚6.15625 disk.md0.out‚ 1480.67578125( disk./.usagePercent‚83.1709997010092 disk.rbd0.out‚ 3.646484375 disk.rbd2.out‚ 3.87109375 disk.sdb.in‚5.796875 disk.sdb1.in‚0 disk.sdb1.out‚0" disk.nvme0n1.out‚ 84179.1484375" disk.sda2.out‚1536.38623046875& proc.memory.availableMB‚ 9601.71875 network.sent‚ 3.1442811e+07 disk.md0p1.in‚ 11.953125 disk.md0p1.out‚ 1480.67578125@wbÞ‘à.‚(:† ²‚èԩ𞆠gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚97.033' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚24.25825 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0AVÔè€á.‚(:õ ²‚èԩл‡ proc.cpu.threads‚17 disk.sdb2.in‚5.796875 disk.sdb.in‚5.796875 disk.sda1.in‚0 network.sent‚ 3.1449174e+07 disk.md0.in‚ 11.953125 disk.rbd2.in‚0! disk.sdb.out‚1537.99072265625 disk.sdb1.out‚0) proc.memory.percent‚18.50731372833252% disk./.usageGB‚2975.0978660583496" disk.sdb2.out‚1537.99072265625 disk.md0.out‚ 1482.2109375 disk.rbd2.out‚ 3.87109375# disk.nvme1n1.in‚201585.86328125! disk.sda.out‚1537.99072265625 disk.dm-0.in‚531780.51171875 disk.sda1.out‚0 disk.rbd0.out‚ 3.646484375 disk.sdb1.in‚0# disk.nvme0n1.out‚84179.19921875$ memory_percent‚41.39423370361328& proc.memory.availableMB‚ 9601.96875# proc.memory.rssMB‚ 3032.23828125 network.recv‚ 4.741686e+06) disk./.usagePercent‚83.18498151018343 disk.md0p1.out‚ 1482.2109375 disk.sda.in‚6.15625! disk.dm-0.out‚161916.50390625 cpu‚25.197600273387813 disk.md0p1.in‚ 11.953125" disk.nvme0n1.in‚330194.6484375" disk.sda2.out‚1537.99072265625" disk.nvme1n1.out‚ 77737.3046875 disk.rbd0.in‚0.015625 disk.sda2.in‚6.15625ÏR&]â.‚(:… Á‚èԘИ‡ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚136.134' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.0335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q\<‚ã.‚(:÷ Á‚èÔËåï‡! disk.sda.out‚1539.47509765625 disk.nvme0n1.out‚ 84179.34375 disk.sda1.in‚0 disk.sda1.out‚0" disk.sdb2.out‚1539.47509765625 disk.md0.out‚ 1483.64453125 disk.rbd2.out‚ 3.87109375# disk.nvme1n1.in‚201585.86328125" disk.nvme0n1.in‚330194.6484375 network.recv‚ 4.746161e+06 disk.rbd2.in‚0 disk.sda.in‚6.15625" disk.sda2.out‚1539.47509765625! disk.sdb.out‚1539.47509765625) proc.memory.availableMB‚ 9601.97265625# disk.nvme1n1.out‚77844.47265625 disk.sdb.in‚5.796875 disk.dm-0.in‚531780.51171875 disk.sda2.in‚6.15625 disk.md0.in‚ 11.953125 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.sdb2.in‚5.796875 disk.rbd0.in‚0.015625 disk.rbd0.out‚ 3.646484375$ memory_percent‚41.39420986175537 cpu‚25.197420767133362 network.sent‚ 3.1455037e+07 disk.sdb1.in‚0! disk.dm-0.out‚162023.81640625 disk.md0p1.out‚ 1483.64453125( disk./.usagePercent‚83.1772990720363% disk./.usageGB‚2974.8231048583984 disk.md0p1.in‚ 11.953125# proc.memory.rssMB‚ 3032.23828125) proc.memory.percent‚18.50731372833252ûpHƒ4ä./ _wandbruntime‚13826  _runtime‚13826f?ŽðËå.Å ' lr-AdamW/pg2‚2.6239775424699697e-06 trainer/global_step‚11699 % _timestamp‚1.7884777732049086e+09 _runtime‚13826.390187082 _step‚959 ' lr-AdamW/pg1‚2.6239775424699697e-06ó7Ðæ.Ê ' lr-AdamW/pg2‚2.6239775424699697e-06 trainer/global_step‚11699 % _timestamp‚1.7884777732049086e+09 _runtime‚13826.390187082 _step‚959 ' lr-AdamW/pg1‚2.6239775424699697e-06¿bz4ç./ _wandbruntime‚13826  _runtime‚13826™Lacò'è.ìO E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚476795418832548 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 222551583 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23250468238336 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 222551583 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚504934687254016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 318136853 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚528576850155008 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚504934663313920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚453611092143984 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 % _timestamp‚1.7884777732066972e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23250468238336 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 318136853 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚528576543984128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23184326688564 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 540688436 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚476711603906560 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _step‚960 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚499961838476288 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 217530707 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚499962090147840 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚13826.392438791 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚685 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚476711621909504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚687 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23184555669648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚499961838476288 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360K‘-±hé.ñO [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23250234569728 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚685 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚476711621909504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚687 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23184555669648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚499961838476288 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚476795418832548 MGDeviceStatsMonitor.on_train_batch_start/reserveõed_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 222551583 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23250468238336 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 222551583 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚504934687254016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 318136853 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚528576850155008 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚504934663313920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚453611092143984 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 % _timestamp‚1.7884777732066972e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23250468238336 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 318136853 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚528576543984128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23184326688564 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 540688436 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚476711603906560 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _step‚960 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚499961838476288 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 217530707 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚499962090147840 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚13826.392438791 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.788477774320379e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 540759775 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 318189915 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚476750907946496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 217548190 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚528620536199680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 222569858 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚476834907678958 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 133026614 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚500003088539136 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 318163545 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 217548887 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23252432277504 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 222569860 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚84522271 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚500003340224000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚528620219556352 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 540733403 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 540759775 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 540733403 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 _step‚961 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 318189915 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 133026616 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚476750889943552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 222569860 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23644164975616 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 222569858 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚504976371224064 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚500003340224000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚697 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚476835153699610›¢kø'ì.úM ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23252432277504 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 222569860 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚84522271 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚500003340224000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚528620219556352 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 540733403 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 540759775 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 540733403 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 _step‚961 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 318189915 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 133026616 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚476750889943552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 222569860 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23644164975616 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 222569858 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚504976371224064 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚500003340224000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚697 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚476835153699610 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.788477774320379e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 540759775 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 318189915 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚476750907946496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 217548190 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚528620536199680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 222569858 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚476834907678958 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 133026614 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚500003088539136 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 318163545 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 217548887Á½³éF4í./ _wandbruntime‚13827  _runtime‚13827ö#øÖ]î.× 0loss/FRt2/assignment_loss‚3.1614692239534286 1loss/SRqqt3/assignment_loss‚0.716553728112062 0loss/SRqqt1/detection_loss‚1.827129390894198 /loss/FRt1/detection_loss‚2.7290510047565806 /loss/FRt3/detection_loss‚1.1448168974012545 0loss/FBt3/detection_loss‚0.49490749350760455 epoch‚9 2loss/SRqqt1/assignment_loss‚1.5414245371140687 _runtime‚13827.509164362 _step‚962 /loss/FRt1/assignment_loss‚2.840862527038112 0loss/FRt3/assignment_loss‚0.9033781077763806 0loss/FBt4/assignment_loss‚0.7430960925477031 1loss/SRqqt4/assignment_loss‚1.300880417227745 &loss/total_loss‚40.804429039443555 $ _timestamp‚1.788477774324262e+09 1loss/FBt2/assignment_loss‚0.17899948481187064 /loss/FRt2/detection_loss‚2.2421465274729555 0loss/SRqqt2/detection_loss‚2.078252725303173 /loss/FBt2/detection_loss‚1.9772820621728897 2loss/SRqqt2/assignment_loss‚1.8276153560727835 0loss/FBt3/assignment_loss‚0.1344326524720427 1loss/SRqqt3/detection_loss‚0.8415203280746937 1loss/SRqqt4/detection_loss‚2.4960311077259205 .loss/FBt1/detection_loss‚2.104003365834554 .loss/FBt4/detection_loss‚2.781116167704264 trainer/global_step‚11699 0loss/FRt4/assignment_loss‚3.4996505117416383 1loss/FBt1/assignment_loss‚0.15355592272613497 /loss/FRt4/detection_loss‚3.0862534070014953¯“ÿ~bï.Ü 0loss/FRt2/assignment_loss‚3.1614692239534286 1loss/SRqqt3/assignment_loss‚0.716553728112062 0loss/SRqqt1/detection_loss‚1.827129390894198 /loss/FRt1/detection_loss‚2.7290510047565806 /loss/FRt3/detection_loss‚1.1448168974012545 0loss/FBt3/detection_loss‚0.49490749350760455 epoch‚9 2loss/SRqqt1/assignment_loss‚1.5414245371140687 _runtime‚13827.509164362 _step‚962 /loss/FRt1/assignment_loss‚2.840862527038112 0loss/FRt3/assignment_loss‚0.9033781077763806 0loss/FBt4/assignment_loss‚0.7430960925477031 1loss/SRqqt4/assignment_loss‚1.300880417227745 &loss/total_loss‚40.804429039443555 $ _timestamp‚1.788477774324262e+09 1loss/FBt2/assignment_loss‚0.17899948481187064 /loss/FRt2/detection_loss‚2.2421465274729555 0loss/SRqqt2/detection_loss‚2.078252725303173 /loss/FBt2/detection_loss‚1.9772820621728897 2loss/SRqqt2/assignment_loss‚1.8276153560727835 0loss/FBt3/assignment_loss‚0.1344326524720427 1loss/SRqqt3/detection_loss‚0.8415203280746937 1loss/SRqqt4/detection_loss‚2.4960311077259205 .loss/FBt1/detection_loss‚2.104003365834554 .loss/FBt4/detection_loss‚2.781116167704264 trainer/global_step‚11699 0loss/FRt4/assignment_loss‚3.4996505117416383 1loss/FBt1/assignment_loss‚0.15355592272613497 /loss/FRt4/detection_loss‚3.0862534070014953ÂZ6Ë{’ð.‚(:‡ ЂèÔÛ›â… gpu.0.gpu‚23.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚141.087' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.27175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ãn§ƒñ.‚(:ø ЂèÔ¹øŠ‡ disk.sdb1.in‚0 disk.dm-0.in‚531780.51171875 disk.dm-0.out‚162210.8359375 disk.sda2.in‚6.15625 disk.sdb2.in‚5.796875 disk.md0.in‚ 11.953125 disk.md0.out‚ 1485.66015625) proc.memory.availableMB‚ 9601.17578125 disk.rbd2.in‚0 disk.rbd2.out‚ 3.87109375 disk.rbd0.in‚0.015625 cpu‚25.197228204273685" disk.sda2.out‚1541.55322265625# disk.nvme1n1.out‚78031.42578125 disk.sda1.out‚0 network.sent‚ 3.1460992e+07) disk./.usagePercent‚83.17732093744539$ disk./.usageGB‚2974.823886871338 disk.sdb1.out‚0! disk.sda.out‚1541.55322265625" disk.nvme0n1.in‚330194.6484375 disk.md0p1.out‚ 1485.66015625# disk.nvme1n1.in‚201585.86328125# disk.nvme0n1.out‚84179.41015625 disk.sda1.in‚0 disk.sdb.in‚5.796875$ memory_percent‚41.39907360076904 proc.cpu.threads‚17 disk.sda.in‚6.15625" disk.sdb2.out‚1541.55322265625) proc.memory.percent‚18.50731372833252 network.recv‚ 4.75014e+06 disk.rbd0.out‚ 3.646484375! disk.sdb.out‚1541.55322265625# proc.memory.rssMB‚ 3032.23828125 disk.md0p1.in‚ 11.953125 ]œ´ò.‚(:„ ß‚èÔÒ®—‡ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚112.572' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.143 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jB]|ó.‚(:ñ ß‚èÔüÒ‰‰ disk.md0p1.out‚ 1487.29296875# proc.memory.rssMB‚ 3032.23828125 disk.sdb.out‚1543.236328125 disk.dm-0.in‚531780.51171875 disk.sda1.out‚0 disk.sda2.in‚6.15625) proc.memory.percent‚18.50731372833252 cpu‚25.197061550946643 network.sent‚ 3.1570184e+07 disk.rbd2.in‚0 disk.sdb.in‚5.796875 disk.sdb1.out‚0" disk.nvme0n1.in‚330194.6484375" disk.nvme0n1.out‚ 84179.5390625! disk.dm-0.out‚162424.16796875 disk.md0.in‚ 11.953125# disk.nvme1n1.in‚201585.86328125$ memory_percent‚41.39389991760254 proc.cpu.threads‚17 disk.md0.out‚ 1487.29296875 disk.sda.out‚1543.236328125) disk./.usagePercent‚83.17796207390398 disk.sdb2.out‚1543.236328125 disk.rbd0.in‚0.015625( proc.memory.availableMB‚ 9602.0234375 disk.sda1.in‚0 disk.sda2.out‚1543.236328125% disk./.usageGB‚2974.8468170166016 disk.sdb2.in‚5.796875 disk.rbd0.out‚ 3.646484375# disk.nvme1n1.out‚78244.62890625 disk.sdb1.in‚0 disk.sda.in‚6.15625 network.recv‚ 4.756321e+06 disk.rbd2.out‚ 3.87109375 disk.md0p1.in‚ 11.953125ùp¯åô.‚(:… î‚èÔ˜»Î… gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚139.868' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.967 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜ê,Dõ.‚(:ð î‚èԉﮇ" disk.nvme0n1.in‚330194.6484375! disk.sda2.out‚1544.9423828125) proc.memory.percent‚18.50731372833252 disk.md0.out‚ 1488.94140625 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 3.87109375# disk.nvme0n1.out‚84179.69921875 disk.sda2.in‚6.15625 disk.sda1.in‚0) disk./.usagePercent‚83.17889780675227 disk.sdb2.in‚5.796875 disk.rbd0.out‚ 3.646484375 disk.sdb.out‚1544.9423828125 disk.sda.out‚1544.9423828125 disk.dm-0.in‚531780.51171875 disk.sdb.in‚5.796875 disk.dm-0.out‚162498.5546875 diíGU7sk.sda1.out‚0$ memory_percent‚41.39747619628906# disk.nvme1n1.in‚201585.86328125 disk.md0.in‚ 11.953125 disk.rbd2.in‚0% proc.memory.availableMB‚ 9601.4375 cpu‚25.196891230156655 proc.cpu.threads‚17 network.recv‚ 4.76111e+06 disk.sdb1.in‚0 disk.md0p1.in‚ 11.953125# proc.memory.rssMB‚ 3032.23828125$ disk./.usageGB‚2974.880283355713 disk.sda.in‚6.15625 disk.md0p1.out‚ 1488.94140625# disk.nvme1n1.out‚78318.85546875 disk.sdb1.out‚0 network.sent‚ 3.1576683e+07! disk.sdb2.out‚1544.9423828125t‹K’ö.‚(:‡ ý‚èÔßô… gpu.0.gpu‚42.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚144.821' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.20525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u‚÷.‚(:÷ ý‚èÔÌ®¤‡ disk.sda2.in‚6.15625 disk.md0p1.out‚ 1490.82421875% memory_percent‚41.397690773010254 network.sent‚ 3.1580113e+07" disk.nvme0n1.in‚330194.6484375" disk.nvme0n1.out‚ 84179.8984375" disk.sda2.out‚1546.89404296875 disk.sdb2.in‚5.796875 network.recv‚ 4.764909e+06 disk.rbd0.in‚0.015625 cpu‚25.19671514568183 disk.dm-0.in‚531780.51171875 disk.rbd2.in‚0 disk.rbd2.out‚ 3.8828125# disk.nvme1n1.in‚201585.86328125! disk.sdb.out‚1546.89404296875 disk.sdb1.in‚0 disk.sda1.in‚0" disk.sdb2.out‚1546.89404296875 disk.md0.in‚ 11.953125 disk.sdb.in‚5.796875# proc.memory.rssMB‚ 3032.23828125( disk./.usagePercent‚83.1850086019586! disk.dm-0.out‚162809.99609375 disk.md0.out‚ 1490.82421875# disk.nvme1n1.out‚78630.09765625 disk.sdb1.out‚0! disk.sda.out‚1546.89404296875) proc.memory.availableMB‚ 9601.40234375) proc.memory.percent‚18.50731372833252$ disk./.usageGB‚2975.098834991455 disk.md0p1.in‚ 11.953125 disk.rbd0.out‚ 3.724609375 disk.sda1.out‚0 disk.sda.in‚6.15625 proc.cpu.threads‚17HWш4ø./ _wandbruntime‚13884  _runtime‚13884J,pNÉù.à & lr-AdamW/pg1‚1.996299956230207e-06 & lr-AdamW/pg2‚1.996299956230207e-06 trainer/global_step‚11749 % _timestamp‚1.7884778308638823e+09 _runtime‚13884.048849536 _step‚963ÐmÎú.È % _timestamp‚1.7884778308638823e+09 _runtime‚13884.048849536 _step‚963 & lr-AdamW/pg1‚1.996299956230207e-06 & lr-AdamW/pg2‚1.996299956230207e-06 trainer/global_step‚11749üEã 4û./ _wandbruntime‚13884  _runtime‚13884 …°µñ'ü.ëO TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 223465333 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚478675905756160 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23348683944960 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 _runtime‚13884.04990854 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚502024607704064 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 319474805 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23348683944960 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325045248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚507018885756416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 223465335 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 319474805 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚530760587979776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 319501149 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚478675923759104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 301105152 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23282493710156 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 133555912 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 223465335 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚502024607704064 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 223465333 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 218415945 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚502024356032512 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚84859322 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚11749 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23348450276352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚84860031 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚709 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23348450276352 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 133555914 _step‚964 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚478675905756160 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚507018861816320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚502024356032512 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚478770120764700 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚455487610015184 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 % _timestamp‚1.7884778308649225e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23282264729072 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚711 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 218415234 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23742027268608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚478769874744256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360¡ÉºZö'ý.ðO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚478675905756160 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚507018861816320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚502024356032512 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚478770120764700 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚455487610015184 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 % _timestamp‚1.7884778308649225e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23282264729072 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚711 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 218415234 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23742027268608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚478769874744256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 223465333 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚478675905756160 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23348683944960 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 _runtime‚13884.04990854 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚502024607704064 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 319474805 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23348683944960 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325045248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚507018885756416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 223465335 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 319474805 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚530760587979776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 319501149 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚478675923759104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 301105152 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23282493710156 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 133555912 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 223465335 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚502024607704064 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 223465333 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 218415945 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚502024356032512 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚84859322 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚11749 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23348450276352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚84860031 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚709 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23348450276352 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 133555914 _step‚964ÄéDZ4þ./ _wandbruntime‚13885  _runtime‚13885‚èFN%ÿ.ôM OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 319501497 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚478715209796096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚478809609611318 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚455525140372608 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 133566716 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23350414302208 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 223483608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚13885.109476006 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚530804578205696 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 223483610 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚507060545786368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23284223218058 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11749 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 319527867 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚662 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚478715191793152 # _timestamp‚1.78847783192049e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23350647984128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚502065606095360 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 218433051 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 319501497 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 319527867 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 223483610 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚84866999 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23744008479232 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 223483608 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 _step‚965 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 218433715 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚507060569726464 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23350647984128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚455525157411968 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 542985105 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚530804269950976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23743724164608 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚478715191793152 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 543011477 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚23284452199350 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚84866337 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMoýiì¬nitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 543011477 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23744008479232 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 223483608 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 _step‚965 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 218433715 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚507060569726464 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23350647984128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚455525157411968 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 542985105 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚530804269950976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23743724164608 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚478715191793152 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 543011477 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚23284452199350 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚84866337 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 543011477 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚478715209796096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚478809609611318 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚455525140372608 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 133566716 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23350414302208 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 223483608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚13885.109476006 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚530804578205696 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 223483610 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚507060545786368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23284223218058 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11749 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 319527867 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚662 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚478715191793152 # _timestamp‚1.78847783192049e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23350647984128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚502065606095360 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 218433051 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 319501497 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 319527867 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 223483610 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292ň 4// _wandbruntime‚13885  _runtime‚13885l*ç_‚/Ù /loss/FRt2/assignment_loss‚3.096161111405021 0loss/FRt3/assignment_loss‚0.9651644855737687 2loss/SRqqt2/assignment_loss‚1.6343505804194138 /loss/FRt1/detection_loss‚2.6228485573083162 trainer/global_step‚11749 _step‚966 0loss/FRt4/assignment_loss‚3.1242226266860964 2loss/SRqqt3/assignment_loss‚0.9120080184779669 1loss/SRqqt4/assignment_loss‚1.915322672413743 1loss/SRqqt2/detection_loss‚2.1851358860731125 .loss/FBt3/detection_loss‚0.519099985739192 0loss/FRt1/assignment_loss‚2.8708777837455273 0loss/FBt4/assignment_loss‚0.3047591979758638 /loss/FRt3/detection_loss‚1.3336726936822136 /loss/FBt2/detection_loss‚1.7826507311118276 1loss/SRqqt1/detection_loss‚1.9675013832747936 .loss/FBt1/detection_loss‚2.657428960005442 _runtime‚13885.115740424 2loss/SRqqt1/assignment_loss‚1.4697180195944384 0loss/FBt1/assignment_loss‚0.1994727760155778 1loss/FBt2/assignment_loss‚0.23684975887798182 /loss/FRt4/detection_loss‚3.1512958717346193 1loss/SRqqt3/detection_loss‚0.7152027495597538 1loss/SRqqt4/detection_loss‚2.6479717506014784 epoch‚9 1loss/FBt3/assignment_loss‚0.07291607740905107 .loss/FBt4/detection_loss‚2.440872781551801 &loss/total_loss‚40.795730134109235 /loss/FRt2/detection_loss‚1.9702256748722375 % _timestamp‚1.7884778319296117e+09767*dƒ/Þ &loss/total_loss‚40.795730134109235 /loss/FRt2/detection_loss‚1.9702256748722375 % _timestamp‚1.7884778319296117e+09 /loss/FRt2/assignment_loss‚3.096161111405021 0loss/FRt3/assignment_loss‚0.9651644855737687 2loss/SRqqt2/assignment_loss‚1.6343505804194138 /loss/FRt1/detection_loss‚2.6228485573083162 trainer/global_step‚11749 _step‚966 0loss/FRt4/assignment_loss‚3.1242226266860964 2loss/SRqqt3/assignment_loss‚0.9120080184779669 1loss/SRqqt4/assignment_loss‚1.915322672413743 1loss/SRqqt2/detection_loss‚2.1851358860731125 .loss/FBt3/detection_loss‚0.519099985739192 0loss/FRt1/assignment_loss‚2.8708777837455273 0loss/FBt4/assignment_loss‚0.3047591979758638 /loss/FRt3/detection_loss‚1.3336726936822136 /loss/FBt2/detection_loss‚1.7826507311118276 1loss/SRqqt1/detection_loss‚1.9675013832747936 .loss/FBt1/detection_loss‚2.657428960005442 _runtime‚13885.115740424 2loss/SRqqt1/assignment_loss‚1.4697180195944384 0loss/FBt1/assignment_loss‚0.1994727760155778 1loss/FBt2/assignment_loss‚0.23684975887798182 /loss/FRt4/detection_loss‚3.1512958717346193 1loss/SRqqt3/detection_loss‚0.7152027495597538 1loss/SRqqt4/detection_loss‚2.6479717506014784 epoch‚9 1loss/FBt3/assignment_loss‚0.07291607740905107 .loss/FBt4/detection_loss‚2.440872781551801Æ=co(Ž„/‚(:ƒ ŒƒèÔº–ˆ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚137.36' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.34 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Zr2|…/‚(:ñ ŒƒèÔÓ鈊 network.recv‚ 4.768549e+06 disk.dm-0.out‚162893.2265625 disk.md0.in‚ 11.953125 disk.sdb.in‚5.796875 disk.sda1.out‚0 network.sent‚ 3.1598154e+07 disk.sda.in‚6.15625# disk.nvme0n1.out‚84179.96484375 disk.md0p1.in‚ 11.953125! disk.sdb2.out‚1548.7216796875% memory_percent‚41.394925117492676 cpu‚25.19656451762123 proc.cpu.threads‚17 disk.sda.out‚1548.7216796875 disk.sdb2.in‚5.796875) proc.memory.availableMB‚ 9601.85546875 disk.sda2.in‚6.15625# disk.nvme1n1.in‚201585.86328125# proc.memory.rssMB‚ 3032.23828125 disk.md0p1.out‚ 1492.578125 disk.rbd0.in‚0.015625 disk.rbd2.out‚ 3.9453125# disk.nvme1n1.out‚78713.26171875% disk./.usageGB‚2975.0989723205566" disk.nvme0n1.in‚330194.6484375 disk.sda1.in‚0) proc.memory.percent‚18.50731372833252) disk./.usagePercent‚83.18501244173775! disk.sda2.out‚1548.7216796875 disk.md0.out‚ 1492.578125 disk.rbd0.out‚ 3.724609375 disk.sdb.out‚1548.7216796875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.dm-0.in‚531780.51171875 disk.rbd2.in‚0Í«Òï†/‚(:… ›ƒèÔ³µú† gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚146.048' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.512 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ž‰J¯}‡/‚(:ò ›ƒèÔàðè‡ disk.md0.in‚ 11.953125 disk.rbd0.out‚ 3.81201171875 network.recv‚ 4.775354e+06 disk.rbd2.out‚ 3.9453125% memory_percent‚41.394877433776855) proc.memory.availableMB‚ 9601.86328125# proc.memory.rssMB‚ 3032.23828125% disk./.usageGB‚2975.0988960266113 disk.sdb1.in‚0 disk.sda1.in‚0 disk.nvme1n1.out‚ 79017.84375) proc.memory.percent‚18.50731372833252 disk.sdb1.out‚0" disk.nvme0n1.in‚330194.6484375 disk.dm-0.in‚531780.51171875 disk.sdb2.in‚5.796875! disk.dm-0.out‚163198.02734375 disk.rbd0.in‚0.015625 disk.sda.in‚6.15625 disk.sda.out‚1550.365234375 disk.sda2.out‚1550.365234375 disk.sdb2.out‚1550.365234375 disk.rbd2.in‚0 disk.sdb.in‚5.796875 network.sent‚ 3.1694598e+07 disk.sdb.out‚1550.365234375 proc.cpu.threads‚17 disk.sda1.out‚0 disk.md0.out‚ 1494.16015625 cpu‚25.196397303926354# disk.nvme1n1.in‚201585.86328125) disk./.usagePercent‚83.18501030852711# disk.nvme0n1.out‚84180.18359375 disk.sda2.in‚6.15625 disk.md0p1.in‚ 11.953125 disk.md0p1.out‚ 1494.160156252€Š–œˆ/‚(:‘ ªƒèÔŸÀɇ gpu.0.gpu‚42.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ï’»†‰/‚(:û ªƒèÔ›­€ˆ network.recv‚ 4.779914e+06" disk.sda2.out‚1552.02685546875 disk.rbd2.out‚ 4.0244140625! disk.sda.out‚1552.02685546875 disk.sdb2.in‚5.796875 proc.cpu.threads‚17 network.sent‚ 3.1700888e+07" disk.nvme1n1.out‚ 79154.3984375 disk.sda1.in‚0 disk.sda2.in‚6.15625 disk.md0p1.out‚ 1495.7578125 disk.rbd2.in‚0# disk.nvme1n1.in‚201585.86328125# disk.nvme0n1.out‚84180.23828125 disk.dm-0.in‚531780.51171875$ memory_percent‚41.39564037322998 cpu‚25.19622133843371! disk.dm-0.out‚163334.63671875 disk.md0.in‚ 11.953125 disk.rbd0.in‚0.015625! disk.sdb.out‚1552.02685546875 disk.md0p1.in‚ 11.953125" disk.sdb2.out‚1552.02685546875% disk./.usageGB‚2975.0988960266113 disk.rbd0.out‚ 3.81201171875 disk.sdb.in‚5.796875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.out‚0 disk.md0.out‚ 1495.7578125) disk./.usagePercent‚83.18501030852711 disk.sda.in‚6.15625" disk.nvme0n1.in‚330194.6484375) proc.memory.availableMB‚ 9601.73828125# proc.memory.rssMB‚ 3032.23828125) proc.memory.percent‚18.507313728332528±'Š/‚(:… ¹ƒèÔÖÌÎ… gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚139.868' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.967 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«L*Žx‹/‚(:í ¹ƒèÔü«ý†! disk.dm-0.out‚163617.59765625 cpu‚25.196050307026557 network.sent‚ 3.1704489e+07$ disk./.usageGB‚2974.823947906494 disk.md0p1.in‚ 11.953125# disk.nvme1n1.in‚201585.86328125# disk.nvme1n1.out‚79437.29296875 disk.sdb.out‚1553.7548828125 disk.sda2.in‚6.15625 disk.sdb1.out‚0" disk.nvme0n1.in‚330194.6484375 disk.sda1.out‚0) disk./.usagePercent‚83.17732264401391 disk.md0.in‚ 11.953125 disk.sdb1.in‚0) proc.memory.availableMB‚ 9601.36328125 disk.sdb.in‚5.796875 disk.dm-0.in‚531780.51171875 disk.rbd0.out‚3.875 disk.rbd2.out‚ 4.0244140625 disk.sda1.in‚0% memory_percent‚41.397929191589355# proc.memory.rssMB‚ 3032.23828125) proc.memory.percent‚18.50731372833252 network.recv‚ 4.78442e+06! disk.sdb2.out‚1553.7548828125 proc.cpu.threads‚17 disk.sda.in‚6.15625" disk.nvme0n1.out‚ 84180.3046875 disk.rbd0.in‚0.015625! disk.sda2.out‚1553.7548828125 disk.sdb2.in‚5.796875 disk.md0.out‚ 1497.421875 disk.rbd2.in‚0 disk.md0p1.out‚ 1497.421875 disk.sda.out‚1553.7548828125Ó€šÕ4Œ// _wandbruntime‚13941  _runtime‚13941G¿ÿÈ/ $ _timestamp‚1.788477888217152e+09 _runtime‚13941.402195338 _step‚967 & lr-AdamW/pg1‚1.454113697775566e-06 & lr-AdamW/pg2‚1.454113697775566e-06 trainer/global_step‚117996¡ÍŽ/Ç & lr-AdamW/pg1‚1.454113697775566e-06 & lr-AdamW/pg2‚1.454113697775566e-06 trainer/global_step‚11799 $ _timestamp‚1.788477888217152e+09 _runtime‚13941.402195338 _step‚967Ç'’¨P4// _wandbruntime‚13941  _runtime‚13941§5@ñ'/ëO YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 134096014 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 224379083 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚11799 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚480640225608704 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23380431750664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚457364144925744 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚504087125260288 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299879424 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23446665982976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚504086873588736 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚532944654713856 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23446899651584 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23841870334464 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 320839101 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚457364127886384 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚480640225608704 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275939328 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 320839101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚657 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚509103060318720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 224379085 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚532944954593280 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚480744330655964 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 219302688 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 545218186 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 $ _timestamp‚1.788477888218201e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚85207331 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23446665982976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚504086873588736 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚532944654713856 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23446899651584 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23841870334464 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 320839101 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚457364127886384 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚480640225608704 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275939328 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 320839101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚657 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚509103060318720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 224379085 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚532944954593280 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚480744330655964 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 219302688 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 545218186 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 $ _timestamp‚1.788477888218201e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚85207331 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚480679511645696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚509144768228864 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 224397358 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚504128375336448 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 320865819 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23448630008832 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 224397360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚667 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚480679493642752 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚480679511645696 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚504128375336448 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23448863690752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 545263179 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚665 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚23448630008832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23843584012288 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788477889414532e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 224397360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚480784065523026 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 219320198 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚532988644944384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚23382390239858 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 134106814 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 320839449 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚504128123651584 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23843876715520 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚457401675283168 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 545263179 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 545236807 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _step‚969 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 545236807 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _runtime‚13942.600883349 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 134106816 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚532988328301056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 320839449 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚504128123651584 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23843876715520 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚457401675283168 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 545263179 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 545236807 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _step‚969 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 545236807 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _runtime‚13942.600883349 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 134106816 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚532988328301056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚480679511645696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚509144768228864 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 224397358 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚504128375336448 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 320865819 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23448630008832 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 224397360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚667 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚480679493642752 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚480679511645696 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚504128375336448 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23448863690752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 545263179 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚665 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚23448630008832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23843584012288 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788477889414532e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 224397360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚480784065523026 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 219320198 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚532988644944384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚23382390239858 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 134106814 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚511187282761216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 134636114 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23545115358208 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚482718786567672 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚459240645757584 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚511187258821120 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23941403201024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚482604527458304 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚11849 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 225292833 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23544881689600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚506149391144960 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 322150709 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 296910848 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23478369791172 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚506149642816512 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 547469888 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 220189997 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚482604509455360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _step‚972 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚506149642816512 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23545115358208 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 134636112 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚482604527458304 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 322177053 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚535128662022144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 $ _timestamp‚1.788477946745981e+09 _runtime‚13999.930986857 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23544881689600Š „ö'¥/ðO JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚482604509455360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _step‚972 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚506149642816512 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23545115358208 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 134636112 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚482604527458304 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 322177053 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚535128662022144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 $ _timestamp‚1.788477946745981e+09 _runtime‚13999.930986857 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23544881689600 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚506149391144960 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚511187282761216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 134636114 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23545115358208 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚482718786567672 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚459240645757584 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚511187258821120 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23941403201024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚482604527458304 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚11849 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 225292833 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23544881689600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚506149391144960 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 322150709 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 296910848 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23478369791172 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚506149642816512 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 547469888 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 220189997 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496Ì ™ø¾4¦// _wandbruntime‚14000  _runtime‚14000ö¬i ‰§/öM A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 220207508 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23547079397376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 547488509 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 225311108 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚511228942791168 trainer/global_step‚11849 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚482643813495296 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚535172655058432 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚459278193154368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚482643813495296 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 220206856 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚973 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 225311110 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 547514881 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚482758521434734 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 134646916 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚535172338415104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚85560592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚23546845715456 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23480099299074 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 547514881 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 322177401 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚459278176115008 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚482643795492352 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 322203771 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 [MDeviceStatsMonitor.on_train_batÀE±ss ch_end/inactive_split_bytes.small_pool.current‚ 292703232 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚506190892892672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 225311110 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚482758275414082 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 322177401 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚506190641207808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚85559942 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 225311108 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23547079397376 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚652 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 % _timestamp‚1.7884779476751895e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 220206856 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚973 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 225311110 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 547514881 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚482758521434734 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 134646916 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚535172338415104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚85560592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚23546845715456 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23480099299074 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 547514881 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 322177401 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚459278176115008 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚482643795492352 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 322203771 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚506190892892672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 225311110 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚482758275414082 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 322177401 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚506190641207808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚85559942 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 225311108 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23547079397376 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚652 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 % _timestamp‚1.7884779476751895e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944ÍÇÊ‘B4©// _wandbruntime‚14000  _runtime‚14000À>y]ª/× 1loss/SRqqt3/detection_loss‚0.9068693847444497 /loss/FBt3/detection_loss‚0.3260986015251646 _step‚974 /loss/FRt4/assignment_loss‚3.593002751469612 1loss/FBt1/assignment_loss‚0.13084320334342578 epoch‚9 0loss/FRt2/assignment_loss‚3.2551477485232883 1loss/SRqqt2/detection_loss‚2.4066657721996307 trainer/global_step‚11849 _runtime‚14000.862141068 0loss/FRt3/assignment_loss‚0.9076471683345263 .loss/FRt3/detection_loss‚1.243637033608518 /loss/FRt4/detection_loss‚3.2413357585668563 $ _timestamp‚1.788477947677437e+09 0loss/FRt1/assignment_loss‚2.7300922982394695 .loss/FRt2/detection_loss‚2.063352206439918 %loss/total_loss‚40.60638794317512 2loss/SRqqt1/assignment_loss‚1.4010440604509533 1loss/SRqqt3/assignment_loss‚0.898695995669665 1loss/FBt3/assignment_loss‚0.05757361570577523 /loss/FBt2/detection_loss‚2.0621454575482536 2loss/SRqqt4/assignment_loss‚1.3475283383158967 1loss/FBt2/assignment_loss‚0.06130936752185288 1loss/SRqqt4/detection_loss‚2.4172591753304005 .loss/FBt4/detection_loss‚2.147004168141972 2loss/SRqqt2/assignment_loss‚1.4386211546936205 0loss/SRqqt1/detection_loss‚2.371267177768656 /loss/FBt1/detection_loss‚2.4487248420715333 0loss/FBt4/assignment_loss‚0.3311971935274249 /loss/FRt1/detection_loss‚2.8193254694342613Ü™%­b«/Ü %loss/total_loss‚40.60638794317512 2loss/SRqqt1/assignment_loss‚1.4010440604509533 1loss/SRqqt3/assignment_loss‚0.898695995669665 1loss/FBt3/assignment_loss‚0.05757361570577523 /loss/FBt2/detection_loss‚2.0621454575482536 2loss/SRqqt4/assignment_loss‚1.3475283383158967 1loss/FBt2/assignment_loss‚0.06130936752185288 1loss/SRqqt4/detection_loss‚2.4172591753304005 .loss/FBt4/detection_loss‚2.147004168141972 2loss/SRqqt2/assignment_loss‚1.4386211546936205 0loss/SRqqt1/detection_loss‚2.371267177768656 /loss/FBt1/detection_loss‚2.4487248420715333 0loss/FBt4/assignment_loss‚0.3311971935274249 /loss/FRt1/detection_loss‚2.8193254694342613 1loss/SRqqt3/detection_loss‚0.9068693847444497 /loss/FBt3/detection_loss‚0.3260986015251646 _step‚974 /loss/FRt4/assignment_loss‚3.593002751469612 1loss/FBt1/assignment_loss‚0.13084320334342578 epoch‚9 0loss/FRt2/assignment_loss‚3.2551477485232883 1loss/SRqqt2/detection_loss‚2.4066657721996307 trainer/global_step‚11849 _runtime‚14000.862141068 0loss/FRt3/assignment_loss‚0.9076471683345263 .loss/FRt3/detection_loss‚1.243637033608518 /loss/FRt4/detection_loss‚3.2413357585668563 $ _timestamp‚1.788477947677437e+09 0loss/FRt1/assignment_loss‚2.7300922982394695 .loss/FRt2/detection_loss‚2.063352206439918Îuû¬/‚(:… „„èÔô¨åˆ gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚144.212' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.053 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å€,€­/‚(:õ „„èÔƒêщ" disk.nvme0n1.in‚330194.6484375 disk.md0.in‚ 11.953125 disk.md0.out‚ 1505.6796875 disk.rbd0.in‚0.015625 disk.rbd2.in‚0# disk.nvme1n1.in‚201585.86328125 disk.sdb1.out‚0" disk.sdb2.out‚1562.34130859375% proc.memory.availableMB‚ 9601.3125) proc.memory.percent‚18.50731372833252 proc.cpu.threads‚17# disk.nvme0n1.out‚84180.95703125 disk.dm-0.out‚ 164622.109375 disk.md0p1.out‚ 1505.6796875 disk.sdb1.in‚0( disk./.usagePercent‚83.1553385217567 disk.dm-0.in‚531780.51171875 disk.sda1.out‚0 disk.sdb2.in‚5.796875 disk.rbd0.out‚ 3.93798828125 disk.sdb.in‚5.796875! disk.sdb.out‚1562.34130859375# proc.memory.rssMB‚ 3032.23828125 network.sent‚ 3.1835913e+07" disk.sda2.out‚1562.34130859375! disk.sda.out‚1562.34130859375$ memory_percent‚41.39823913574219% disk./.usageGB‚2974.0376892089844 disk.sda1.in‚0# disk.nvme1n1.out‚80441.15234375 disk.sda.in‚6.15625 network.recv‚ 4.808456e+06 disk.sda2.in‚6.15625 disk.md0p1.in‚ 11.953125 disk.rbd2.out‚ 4.150390625 cpu‚25.195064483173244žÄ­®/‚(:… “„èÔÿ¨þ… gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚142.93' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.7325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®(}éy¯/‚(:î “„èÔõÐõ‡ disk.dm-0.in‚531780.51171875 disk.sda2.in‚6.15625 disk.rbd2.in‚0 disk.sdb1.in‚0) proc.memory.availableMB‚ 9601.51171875 network.recv‚ 4.814977e+06! disk.dm-0.out‚164791.48046875 disk.sdb2.in‚5.796875 disk.rbd0.in‚0.015625# disk.nvme1n1.in‚201585.86328125$ memory_percent‚41.39702320098877 disk.sdb2.out‚ 1564.05078125 cpu‚25.194862911694457 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda2.out‚ 1564.05078125 disk.rbd0.out‚ 3.93798828125# disk.nvme1n1.out‚80610.46484375 network.sent‚ 3.1945026e+07( disk./.usagePercent‚83.1476415777772 disk.md0p1.out‚ 1507.32421875 disk.sdb1.out‚0 disk.sda1.in‚0$ disk./.usageGB‚2973.762409210205 disk.md0p1.in‚ 11.953125 disk.md0.in‚ 11.953125 disk.md0.out‚ 1507.32421875 disk.rbd2.out‚ 4.150390625 disk.sdb.in‚5.796875 disk.sda.out‚ 1564.05078125) proc.memory.percent‚18.50731372833252 disk.sdb.out‚ 1564.05078125 disk.sda.in‚6.15625# proc.memory.rssMB‚ 3032.23828125" disk.nvme0n1.in‚330194.6484375! disk.nvme0n1.out‚ 84181.015625(V<¯°/‚(:… ¢„èÔá¢Ç‡ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚36.0 gpu.0.powerWatts‚139.868' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.967 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ±ÝÊ{±/‚(:𠢄èÔ𜦈 disk.dm-0.in‚531780.51171875 disk.sdb2.out‚1566.064453125 disk.md0.in‚ 11.953125 disk.md0.out‚ 1509.27734375 disk.rbd2.out‚ 4.150390625 disk.sda.in‚6.15625) proc.memory.percent‚18.50731372833252 cpu‚25.19467145450319! disk.dm-0.out‚164834.58984375 disk.sdb1.in‚0" disk.nvme0n1.in‚330194.6484375) proc.memory.availableMB‚ 9601.51171875 disk.md0p1.in‚ 11.953125 disk.rbd2.in‚0# disk.nvme1n1.out‚80653.37109375 disk.sdb1.out‚0 network.recv‚ 4.819776e+06) disk./.usagePercent‚83.14763709803485 disk.sdb2.in‚5.796875# proc.memory.rssMB‚ 3032.23828125 proc.cpu.threads‚17 disk.md0p1.out‚ 1509.27734375 disk.sdb.in‚5.796875 network.sent‚ 3.1951423e+07 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0.015625 disk.sdb.out‚1566.064453125 disk.nvme0n1.out‚ 84181.21875 disk.sda2.in‚6.15625 disk.rbd0.out‚ 3.93798828125# disk.nvme1n1.in‚201585.86328125$ memory_percent‚41.39702320098877# disk./.usageGB‚2973.76224899292 disk.sda2.out‚1566.064453125 disk.sda.out‚1566.064453125Yë4²// _wandbruntime‚14050  _runtime‚14050îù&›É³/à % _timestamp‚1.7884779972780392e+09 _runtime‚14050.463346992 _step‚975 & lr-AdamW/pg1‚6.266492876242654e-07 & lr-AdamW/pg2‚6.266492876242654e-07 trainer/global_step‚118999)íδ/È & lr-AdamW/pg1‚6.266492876242654e-07 & lr-AdamW/pg2‚6.266492876242654e-07 trainer/global_step‚11899 % _timestamp‚1.7884779972780392e+09 _runtime‚14050.463346992 _step‚975Ϫ‰À4µ// _wandbruntime‚14050  _runtime‚14050ܶ8?ñ'¶/ëO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 549721590 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 549695244 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884779972795415e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 226206583 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 323488661 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 323515005 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚508211908701184 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚484568811304960 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23643097396224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚24041519215616 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚484693488499824 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 221077635 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 135176214 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚461117180668144 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23576078850596 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚484568811304960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 221078331 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚85902117 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23576307831680 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚513271481263616 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚694 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚508211908701184 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚11899 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 323515005 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚6961XqÕ·/ðO G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚513271481263616 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚694 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚508211908701184 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚11899 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 323515005 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚696 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 549721590 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 549695244 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884779972795415e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 226206583 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 323488661 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 323515005 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚508211908701184 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚484568811304960 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23643097396224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚24041519215616 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚484693488499824 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 221077635 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 135176214 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚461117180668144 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23576078850596 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚484568811304960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 221078331 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚85902117 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23576307831680 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496Ð"ˆxC4¸// _wandbruntime‚14051  _runtime‚14051Á†¶yú&¹/ôM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚513313141293568 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚461154711025568 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _step‚977 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 549740211 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 323541723 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 323515353 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23645295104000 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚513313165233664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 549740211 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 226224860 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 549766583 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚23645061422080 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 221095442 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚537356678256640 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚24043216125440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 226224858 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚484608097341952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 323515353 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 135187016 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚23578266320874 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 # _timestamp‚1.78847799853179e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚508253410448896 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚508253158764032 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23645295104000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚537356357419008 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚461154693986208 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23578037339582 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚508253410448896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚85908428 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚484732977346442 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 549766583 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 226224858 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 221096126 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 226224860 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _runtime‚14051.718109805 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 323541723 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 549740211 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 323541723 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 323515353 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23645295104000 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚513313165233664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 549740211 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 226224860 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 549766583 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚23645061422080 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 221095442 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚537356678256640 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚24043216125440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 226224858 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚484608097341952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 323515353 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 135187016 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚23578266320874 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 # _timestamp‚1.78847799853179e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚508253410448896 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚508253158764032 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23645295104000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚537356357419008 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚461154693986208 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23578037339582 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚508253410448896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚85908428 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚484732977346442 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 549766583 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 226224858 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 221096126 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 226224860 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _runtime‚14051.718109805 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 323541723 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 324852957 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23741313102848 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 551973292 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 227120333 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚486667944411532 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚486533131157504 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚539496633672704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23674016891104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 221971339 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚486533113154560 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 227120335 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23741313102848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚510274677928960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚486533131157504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚86255025 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚510274677928960 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 221970668 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚24141266369024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚486667698391088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _runtime‚14103.821086057 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23741546771456 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 324852957 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚11949 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 324826613 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 551946946 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚510274426257408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚671 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚515355655825920 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23741546771456 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚510274426257408 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0²{®û÷'Ë/ñO QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 227120333 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚486667944411532 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚486533131157504 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚539496633672704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23674016891104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 221971339 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚486533113154560 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 227120335 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23741313102848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚510274677928960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚486533131157504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚86255025 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚510274677928960 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 221970668 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚24141266369024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚486667698391088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _runtime‚14103.821086057 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23741546771456 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 324852957 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚11949 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 324826613 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 551946946 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚510274426257408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚671 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚515355655825920 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23741546771456 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚510274426257408 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚486533113154560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚515355679766016 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 227120333 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 324826613 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 227120335 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 135716312 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 324852957 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23741313102848 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 551973292 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0ÔÁ¿«å4Ì// _wandbruntime‚14104  _runtime‚14104îKz(ëÍ/öM \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚539540624947200 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 135727114 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 552018285 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚670 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚23676204361382 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23743277128704 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚515397363736064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _runtime‚14104.759807189 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚23743277128704 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 trainer/global_step‚11949 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚86261785 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚515397339795968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚463031211857408 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 227138610 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 551991913 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 324879675 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚486572417194496 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 221988229 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚510315928005120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23675975380090 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23743510810624 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23743510810624 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚486707433258150 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚486707187237498 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 _step‚981 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 227138610 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚510315928005120 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 324879675 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚86261115 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚24143261211136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pꯗÌ ool.peak‚ 323478528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚672 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 551991913 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚539540314595328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚486572399191552 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 324853305 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582¼„·²'Î/ûM \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚539540624947200 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 135727114 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 552018285 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚670 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚23676204361382 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23743277128704 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚515397363736064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _runtime‚14104.759807189 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚23743277128704 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 trainer/global_step‚11949 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚86261785 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚515397339795968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚463031211857408 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 227138610 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 551991913 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 324879675 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚486572417194496 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 221988229 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚510315928005120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23675975380090 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23743510810624 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23743510810624 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚486707433258150 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚486707187237498 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 _step‚981 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 227138610 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚510315928005120 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 324879675 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚86261115 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚24143261211136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚672 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 551991913 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚539540314595328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚486572399191552 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 324853305 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582Õ5­™`4Ï// _wandbruntime‚14104  _runtime‚14104Ï $]Ð/× /loss/FBt3/detection_loss‚0.6354210573990713 2loss/SRqqt1/assignment_loss‚1.3540475892321187 2loss/SRqqt3/assignment_loss‚0.8132711602064371 .loss/FRt3/detection_loss‚1.732624766106407 .loss/FBt1/detection_loss‚2.342392547564073 $ _timestamp‚1.788478051576148e+09 0loss/FRt3/assignment_loss‚1.1871112197637559 0loss/FRt4/assignment_loss‚3.7199056473645298 /loss/FRt1/detection_loss‚2.6605911237352036 trainer/global_step‚11949 2loss/SRqqt4/assignment_loss‚1.4290871016681195 1loss/FBt1/assignment_loss‚0.24459054931446644 /loss/FRt2/detection_loss‚1.8681526963337092 0loss/SRqqt4/detection_loss‚2.520714321732521 epoch‚9 _runtime‚14104.761036554 /loss/FRt1/assignment_loss‚2.666638011441511 /loss/FRt2/assignment_loss‚3.433609896472522 0loss/FBt3/assignment_loss‚0.0336519074041266 .loss/FRt4/detection_loss‚2.861899318117084 1loss/SRqqt3/detection_loss‚0.9343550896521696 /loss/FBt2/detection_loss‚2.1178404174067755 2loss/SRqqt2/assignment_loss‚1.2257139632877494 1loss/FBt2/assignment_loss‚0.17027357106979288 .loss/FBt4/detection_loss‚2.942093897610903 _step‚982 2loss/FBt4/assignment_loss‚0.024159646418410397 %loss/total_loss‚40.89070280285418 1loss/SRqqt2/detection_loss‚2.4565938051257814 1loss/SRqqt1/detection_loss‚1.5159634984269434:"Á7bÑ/Ü /loss/FRt1/assignment_loss‚2.666638011441511 /loss/FRt2/assignment_loss‚3.433609896472522 0loss/FBt3/assignment_loss‚0.0336519074041266 .loss/FRt4/detection_loss‚2.861899318117084 1loss/SRqqt3/detection_loss‚0.9343550896521696 /loss/FBt2/detection_loss‚2.1178404174067755 2loss/SRqqt2/assignment_loss‚1.2257139632877494 1loss/FBt2/assignment_loss‚0.17027357106979288 .loss/FBt4/detection_loss‚2.942093897610903 _step‚982 2loss/FBt4/assignment_loss‚0.024159646418410397 %loss/total_loss‚40.89070280285418 1loss/SRqqt2/detection_loss‚2.4565938051257814 1loss/SRqqt1/detection_loss‚1.5159634984269434 /loss/FBt3/detection_loss‚0.6354210573990713 2loss/SRqqt1/assignment_loss‚1.3540475892321187 2loss/SRqqt3/assignment_loss‚0.8132711602064371 .loss/FRt3/detection_loss‚1.732624766106407 .loss/FBt1/detection_loss‚2.342392547564073 $ _timestamp‚1.788478051576148e+09 0loss/FRt3/assignment_loss‚1.1871112197637559 0loss/FRt4/assignment_loss‚3.7199056473645298 /loss/FRt1/detection_loss‚2.6605911237352036 trainer/global_step‚11949 2loss/SRqqt4/assignment_loss‚1.4290871016681195 1loss/FBt1/assignment_loss‚0.24459054931446644 /loss/FRt2/detection_loss‚1.8681526963337092 0loss/SRqqt4/detection_loss‚2.520714321732521 epoch‚9 _runtime‚14104.761036554ÖÊøA§‘Ò/‚(:† í„èÔšéý… gpu.0.gpu‚26.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚129.614' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.4035 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü‹È €Ó/‚(:õ í„èÔí·" disk.nvme0n1.in‚330194.6484375 disk.rbd0.out‚ 3.93798828125 disk.rbd2.in‚0$ memory_percent‚41.40059947967529 disk.md0p1.in‚ 11.953125 disk.md0.in‚ 11.953125 disk.md0.out‚ 1517.60546875 disk.rbd2.out‚ 4.150390625 cpu‚25.193714564272433# disk.nvme1n1.out‚81031.31640625 disk.sdb1.in‚0 disk.dm-0.in‚531780.51171875 disk.sda1.out‚0 disk.sda2.in‚6.15625 network.sent‚ 3.2079644e+07 network.recv‚ 4.842937e+06 disk.sda.in‚6.15625 disk.sda.out‚1574.6767578125 disk.md0p1.out‚ 1517.60546875) disk./.usagePercent‚83.15684648835983 disk.sdb.in‚5.796875 disk.dm-0.out‚ 165399.1875! disk.sda2.out‚1574.6767578125 disk.sdb2.in‚5.796875# proc.memory.rssMB‚ 3032.23828125$ disk./.usageGB‚2974.091621398926# disk.nvme1n1.in‚201585.86328125 disk.sda1.in‚0! disk.sdb2.out‚1574.6767578125) proc.memory.availableMB‚ 9600.92578125) proc.memory.percent‚18.50731372833252 disk.sdb.out‚1574.6767578125 disk.sdb1.out‚0# disk.nvme0n1.out‚84367.87109375 disk.rbd0.in‚0.015625 proc.cpu.threads‚17à œÔ/‚(:‘ ü„èÔžýŇ gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚34.0 gpu.0.powerWatts‚104.161' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚26.040249999999997 gpu.0.smClock‚1215 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œîؤ‚Õ/‚(:÷ ü„èÔ”þ‡ network.sent‚ 3.2189551e+07 disk.md0p1.in‚ 11.953125 disk.md0.out‚ 1519.1484375 disk.rbd0.in‚0.015625) proc.memory.percent‚18.50731372833252 cpu‚25.193523834411945) disk./.usagePercent‚83.14932617422153) proc.memory.availableMB‚ 9601.00390625" disk.sda2.out‚1576.28662109375 disk.rbd2.out‚ 4.150390625 disk.sda.in‚6.15625" disk.nvme0n1.in‚330194.6484375# disk.nvme0n1.out‚84374.09765625 network.recv‚ 4.849743e+06 disk.dm-0.in‚531780.51171875 disk.sda1.out‚0 disk.sdb2.in‚5.796875# disk.nvme1n1.out‚81209.52734375! disk.sda.out‚1576.28662109375 proc.cpu.threads‚17 disk.md0.in‚ 11.953125 disk.rbd0.out‚ 3.93798828125# proc.memory.rssMB‚ 3032.23828125 disk.sda1.in‚0" disk.sdb2.out‚1576.28662109375 disk.sdb.in‚5.796875 disk.sdb1.in‚0$ memory_percent‚41.40012264251709% disk./.usageGB‚2973.8226585388184 disk.dm-0.out‚ 165583.625 disk.sda2.in‚6.15625 disk.rbd2.in‚0! disk.sdb.out‚1576.28662109375 disk.md0p1.out‚ 1519.1484375# disk.nvme1n1.in‚201585.86328125 disk.sdb1.out‚0ßW|Ö/‚(:ñ ‹…èÔ³ŠŒ‡ network.sent‚ 3.2192605e+07 network.recv‚ 4.853401e+06 disk.dm-0.in‚531780.51171875 disk.sda2.in‚6.15625 proc.memory.availableMB‚9601# proc.memory.rssMB‚ 3032.23828125 disk.sdb.in‚5.796875! disk.nvme0n1.out‚ 84374.140625 disk.sda1.out‚0 disk.md0p1.in‚ 11.953125% disk./.usageGB‚2973.7784690856934 disk.rbd2.in‚0! disk.sda.out‚1577.90283203125" disk.nvme0n1.in‚330194.6484375" disk.sda2.out‚1577.90283203125 disk.md0.in‚ 11.953125 disk.md0.out‚ 1520.69921875! disk.sdb.out‚1577.90283203125 disk.sda1.in‚0 disk.md0p1.out‚ 1520.69921875 proc.cpu.threads‚17 disk.sdb2.in‚5.796875 disk.rbd0.in‚0.015625# disk.nvme1n1.in‚201585.86328125" memory_percent‚41.400146484375) proc.memory.percent‚18.50731372833252 disk.rbd0.out‚ 3.93798828125 disk.rbd2.out‚ 4.150390625# disk.nvme1n1.out‚81252.92578125 disk.sdb1.out‚0 disk.sda.in‚6.15625) disk./.usagePercent‚83.14809061861743! disk.dm-0.out‚165627.06640625" disk.sdb2.out‚1577.90283203125 disk.sdb1.in‚0 cpu‚25.193304789259116ôÈ.‘×/‚(:† ‹…èÔÄÞ°‡ gpu.0.gpu‚4.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚102.935' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.73375 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çRØ/‚(:… š…èÔ¨Úò… gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚33.0 gpu.0.powerWatts‚103.544' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚25.886 gpu.0.smClock‚1200 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c·±ÉÙ/‚(:ö š…èÔ£Ÿ‡ disk.nvme0n1.out‚ 84515.1875# disk.nvme1n1.in‚201585.86328125! disk.sdb.out‚1580.27197265625 disk.sda.in‚6.15625! disk.dm-0.out‚165768.44140625" disk.sda2.out‚1580.27197265625" disk.sdb2.out‚1580.27197265625 disk.md0.in‚ 11.953125) proc.memory.percent‚18.50731372833252 network.recv‚ 4.858154e+06 disk.rbd2.out‚ 4.150390625 disk.sda1.out‚0 disk.sdb2.in‚5.796875( proc.memory.availableMB‚ 9601.1171875 disk.rbd0.out‚ 3.93798828125 disk.sdb1.out‚0 disk.sda1.in‚0 cpu‚25.193117568931324) disk./.usagePercent‚83.15702269155891 disk.rbd2.in‚0 disk.sdb1.in‚0 disk.dm-0.in‚531780.51171875 network.sent‚ 3.2199081e+07 disk.rbd0.in‚0.015625! disk.sda.out‚1580.27197265625" disk.nvme0n1.in‚330194.6484375 disk.md0.out‚ 1522.984375# proc.memory.rssMB‚ 3032.23828125 disk.sdb.in‚5.796875 disk.md0p1.out‚ 1522.984375 proc.cpu.threads‚17 disk.sda2.in‚6.15625 disk.md0p1.in‚ 11.953125% memory_percent‚41.399431228637695% disk./.usageGB‚2974.0979232788086# disk.nvme1n1.out‚81253.25390625?¨ø4Ú// _wandbruntime‚14161  _runtime‚14161éW´‰ËÛ/Å _step‚983 ' lr-AdamW/pg1‚1.4227404545077534e-07 ' lr-AdamW/pg2‚1.4227404545077534e-07 trainer/global_step‚11999 % _timestamp‚1.7884781083610606e+09 _runtime‚14161.546065478Xq­ÐÜ/Ê ' lr-AdamW/pg2‚1.4227404545077534e-07 trainer/global_step‚11999 % _timestamp‚1.7884781083610606e+09 _runtime‚14161.546065478 _step‚983 ' lr-AdamW/pg1‚1.4227404545077534e-07×Ãã4Ý// _wandbruntime‚14161  _runtime‚14161•ð´$ò'Þ/ìO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚488642154302796 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚24240783868928 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚512336943813632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 136256412 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 228034085 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚86599342 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚464870199371184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚488497433007104 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚512337195485184 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 % _timestamp‚1.7884781083625703e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 554198648 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23839528809472 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚488497433007104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚706 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 222855050 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚86598638 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚517439878268416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚488642400323240 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚464870216410544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚24241076585472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 326164565 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 _step‚984 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚512337195485184 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚541680954853888 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚517439854328320 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23771954931612 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 136256414 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 228034083 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚512336943813632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚24240783868928 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚512336943813632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 136256412 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 228034085 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚86599342 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚464870199371184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚488497433007104 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚512337195485184 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 % _timestamp‚1.7884781083625703e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 554198648 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23839528809472 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚488497433007104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚706 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 222855050 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚86598638 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚517439878268416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚488642400323240 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚464870216410544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚24241076585472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 326164565 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 _step‚984 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚512337195485184 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚541680954853888 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚517439854328320 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23771954931612 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 136256414 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 228034083 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚512336943813632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚488536701041152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚11999 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚24242802186240 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚488536719044096 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚512378445561344 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 228052358 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚14162.603215494 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23773913420598 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 228052358 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚464907729728608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚86605293 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 554243615 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚541724661322240 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚517481562238464 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚488536719044096 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 326191257 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚670 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚488681889169858 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 554269987 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 222873179 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚86605963 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚672 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23841492835328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚488536701041152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 222872507 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚464907746767968 $ _timestamp‚1.788478109417914e+09 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 136267214 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 326217627 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582¶©ù'â/úM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚23774142401890 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 222872507 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚464907746767968 $ _timestamp‚1.788478109417914e+09 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 136267214 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 326217627 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚488536701041152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚11999 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚24242802186240 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚488536719044096 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚512378445561344 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 228052358 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚14162.603215494 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23773913420598 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 228052358 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚464907729728608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚86605293 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 554243615 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚541724661322240 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚517481562238464 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚488536719044096 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 326191257 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚670 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚488681889169858 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 554269987 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 222873179 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚86605963 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚672 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23841492835328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚488536701041152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚514399461369856 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚24340588034560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚466746734281744 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚514399713041408 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 327528861 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚514399713041408 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 228947835 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚490461716853760 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 556476696 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 223745095 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 327528861 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚543864640865280 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚519524052830720 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23937744516096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚519524076770816 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _runtime‚14219.189320348 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚490461734856704 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚86949254 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23869892972120 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23937744516096 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 327502517 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23937978184704 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318753792 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚514399461369856 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 228947833 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 327502517 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _step‚988 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 136796514 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚490616856234948 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚490616610214504 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 228947835 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 228947833 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23937978184704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚490461734856704 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2wHçö'ñ/ðO _step‚988 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 136796514 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚490616856234948 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚490616610214504 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 228947835 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 228947833 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23937978184704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚490461734856704 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 223745768 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23870121953204 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 556450350 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294813696 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 136796512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚514399461369856 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚24340588034560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚466746734281744 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚514399713041408 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 327528861 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚514399713041408 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 228947835 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚490461716853760 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 556476696 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 223745095 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 327528861 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚543864640865280 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚519524052830720 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23937744516096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚519524076770816 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _runtime‚14219.189320348 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚490461734856704 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚86949254 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23869892972120 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23937744516096 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 327502517 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23937978184704 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318753792 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚514399461369856 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 228947833 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 327502517 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309ܲ³§œò/‚(:‘ Ö…èÔ½¹† gpu.0.gpu‚47.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.948745727539062) gpu.0.memoryAllocatedBytes‚ 7686914048 gpu.0.temp‚35.0 gpu.0.powerWatts‚103.039' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚25.759749999999997 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01ßlÄ…ó/‚(:ú Ö…èÔ󢈈 disk.rbd0.out‚ 4.00048828125! disk.sdb.out‚1586.81298828125 disk.sda.in‚6.15625' proc.memory.availableMB‚ 9600.703125# proc.memory.rssMB‚ 3032.23828125) disk./.usagePercent‚83.18496796429585 disk.rbd0.in‚0.015625" disk.nvme1n1.out‚ 81254.5546875 disk.sdb.in‚5.796875" disk.nvme0n1.in‚330194.6484375) proc.memory.percent‚18.50731372833252" disk.sda2.out‚1586.81298828125 disk.sdb2.in‚5.796875" disk.sdb2.out‚1586.81298828125 disk.md0.out‚ 1529.27734375 disk.rbd2.in‚0 disk.rbd2.out‚ 4.150390625 proc.cpu.threads‚17 network.sent‚ 3.2337307e+07 network.recv‚ 4.878073e+06 disk.sdb1.in‚0! disk.sda.out‚1586.81298828125 disk.sda1.in‚0 disk.md0p1.in‚ 11.953125 disk.md0p1.out‚ 1529.27734375# disk.nvme0n1.out‚85454.33984375! disk.dm-0.out‚166708.89453125 disk.md0.in‚ 11.953125$ disk./.usageGB‚2975.097381591797 disk.sdb1.out‚0 disk.sda2.in‚6.15625$ memory_percent‚41.40195846557617# disk.nvme1n1.in‚201585.86328125 disk.dm-0.in‚531780.51171875 disk.sda1.out‚0 cpu‚25.192354220911202ÕÒ¤A4ô// _wandbruntime‚14220  _runtime‚14220c¨f–õ/õM OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 228966110 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚490656099060914 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚466784247599808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚543908317116416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 223763034 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚514440711432704 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚490501020893696 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚678 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 327529209 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚676 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚519565760740864 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 327555579 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚490501002890752 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23939942223872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 556495317 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 556521689 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚86955720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚519565736800768 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚514440963117568 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 556521689 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 $ _timestamp‚1.788478166996901e+09 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚490656345081566 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚466784264639168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23939708541952 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚24342877213184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 556495317 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23939942223872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 228966108 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚543908637954048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚490501002890752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚514440963117568 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 556521689 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 $ _timestamp‚1.788478166996901e+09 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚490656345081566 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚466784264639168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23939708541952 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚24342877213184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 556495317 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23939942223872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 228966108 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚543908637954048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚490501002890752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 223763034 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚514440711432704 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚490501020893696 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚678 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 327529209 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚676 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚519565760740864 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 327555579 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚490501002890752 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23939942223872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 556495317 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 556521689 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚86955720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚519565736800768 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 PDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193815040 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚390 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 157635072 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚24041745933312 ^PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚ 137811814 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚309 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 181575168 % _timestamp‚1.7884782725826051e+09 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 347418624 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚469537884728736 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 171820472 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 7134511104 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 225324468 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6458321920 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚517587232529408 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3532 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 329040358 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6741059457 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚24041570121216 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 329061069 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚522720845469184 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚493545662408192 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚582 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚493545680411136 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 559511897 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 559491184 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚517587426344448 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚24041570121216 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1319 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚273 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚517587232529408 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6966900736 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 175812096 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 565025792 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3532 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚87512654 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20711 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 7134511104 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 175812096 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 560288777 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 572522496 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚273 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚493511295529590Q+=Õ^*Š0ØT E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚24041745933312 ^PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚ 137811814 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚309 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 181575168 % _timestamp‚1.7884782725826051e+09 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 347418624 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚469537884728736 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 171820472 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 7134511104 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 225324468 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6458321920 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚517587232529408 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3532 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 329040358 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6741059457 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚24041570121216 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 329061069 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚522720845469184 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚493545662408192 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚582 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚493545680411136 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 559511897 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 559491184 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚517587426344448 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚24041570121216 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1319 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚273 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚517587232529408 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6966900736 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 175812096 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 565025792 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3532 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚87512654 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20711 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 7134511104 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 175812096 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 560288777 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 572522496 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚273 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚493511295529590 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚469537901768096 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 _step‚991 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20711 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚23973582621326 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32006 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚493545680411136 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1300 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚24446032322048 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 230450828 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 559511897 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚547166877791232 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚273 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 329061069 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚493545662408192 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚24041745933312 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188859832 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 230450826 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚87512266 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚309 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚24445874686976 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20713 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚ 137811812 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚388 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193815040 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚390 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 157635072 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6561988608ßů4‹0/ _wandbruntime‚14326  _runtime‚14326 ½òb)Œ0ÜR \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚493537516246866 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32006 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚24042043981312 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚87518229 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884782737121606e+09 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚522751207222784 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚273 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚547197531829760 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚522751183282688 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 347418624 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 175812096 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚36 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚469563461417728 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 559515474 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 225341725 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 181575168 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 193815040 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188859832 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 572522496 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 329069977 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 193815040 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚24042219793408 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚24042043981312 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 157635072 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 559536187 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚309 trainer/global_step‚12089 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 329049266 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚23973883008666 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚309 _runtime‚14326.89726714 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚517617772063744 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1300 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 7134511104 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 230466208 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚87517841 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 175812096 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚493575728082432 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3532 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 329069977 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1319 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 565025792 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6561988608 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 572522496 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚493575728082432 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 6458321920 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚24042219793408 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚388 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 7134511104 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚517617965878784 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 565025792 \NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚ 137823496 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 230466210 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚517617772063744 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 230466208 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 _step‚992 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 225341335 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6741059457 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 171820472 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚24446506182144 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚493537327387034 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚273 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚273 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 559536187 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚582 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 230466210 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚390 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚309 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 329049266 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚23974054829138 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3532pÒµèm0áR M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 559515474 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚547197713404928 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚469563444378368 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 560288777 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20711 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚517617965878784 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 347418624 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 175812096 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚36 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚469563461417728 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 559515474 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 225341725 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 181575168 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 193815040 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188859832 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 572522496 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 329069977 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 193815040 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚24042219793408 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚24042043981312 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 157635072 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 559536187 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚309 trainer/global_step‚12089 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 329049266 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚23973883008666 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚309 _runtime‚14326.89726714 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚517617772063744 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1300 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 7134511104 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 230466208 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚87517841 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 175812096 _KDeviceStatsMonitor.on_validatioo+þún_batch_end/allocated_bytes.large_pool.freed‚493575728082432 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3532 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 329069977 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1319 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 565025792 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6561988608 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 572522496 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚493575728082432 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 6458321920 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚24042219793408 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚388 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 7134511104 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚517617965878784 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 565025792 \NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚ 137823496 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 230466210 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚517617772063744 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 230466208 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 _step‚992 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 225341335 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6741059457 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 171820472 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚24446506182144 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚493537327387034 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚273 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚273 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 559536187 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚582 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 230466210 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚390 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚309 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 329049266 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚23974054829138 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3532 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚493537516246866 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32006 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚24042043981312 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚87518229 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884782737121606e+09 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚522751207222784 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚273 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚547197531829760 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚522751183282688 B&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.564504086971283 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0011337868636474013 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.022392291575670242 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 <"Purity/2FRt1SRqqt1FBt/event_purity‚0.0059523810632526875 particle/accuracy_1_of_3‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17354938387870789 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9895833134651184 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9907757043838501 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5541612505912781 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1226053535938263 (particle/accuracy‚0.8581196069717407 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.026889055967330933 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5301781296730042 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23722390830516815 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.44767966866493225 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11926806718111038 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2023809552192688 +jet/accuracy_4_of_5‚0.02264465019106865 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5017858147621155 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.02380952425301075 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 particle/accuracy_4_of_7‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5564802289009094 +jet/accuracy_2_of_3‚0.27623873949050903 particle/accuracy_2_of_3‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.5057239532470703 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 epoch‚9 8"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.563230037689209 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5785521864891052 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5206280946731567 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.05349935591220856 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9322916865348816 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2938079237937927 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14555546641349792 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.38154762983322144 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5672968626022339 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1231442391872406 6 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.270114928483963 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0833333358168602 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11154208332300186 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.327725887298584 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_6‚1 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7348484396934509 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.019607843831181526 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5603302121162415 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12336037307977676 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2641419470310211 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4469697177410126 particle/accuracy_4_of_4‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983899593353271 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.12381001561880112 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974507093429565 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.5093567371368408 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.19949495792388916 particle/accuracy_3_of_4‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9910677671432495 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5357142686843872 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0015873016091063619 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9876037240028381 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.6987180113792419 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.992834210395813 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2014327198266983 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.5775985717773438 particle/accuracy_1_of_4‚1 6validation_average_jet_accuracy‚0.4586603045463562 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1846516728401184 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5468412041664124 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.28956836462020874 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.40640079975128174 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01189887523651123 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ) Purity/3FRt1SRqqt*FBt/FRt_purity‚0.25 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.16023391485214233 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.46666663885116577 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1626984179019928 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9917800426483154 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6681765913963318 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 8"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.584971010684967 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6360118985176086 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6491227746009827 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.4027777910232544 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15321384370326996 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.20802128314971924 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.33982861042022705 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.04784398153424263 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2014891654253006 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5534487366676331 *jet/accuracy_1_of_3‚0.8892883062362671 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.06711822748184204 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.18453028798103333 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.019607843831181526 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3046554625034332 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5947150588035583 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.041239891201257706 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2233007699251175 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4423363208770752 +jet/accuracy_3_of_4‚0.07943448424339294 _runtime‚14343.392206107 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.17178021371364594 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.30482909083366394 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5343220233917236 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 6 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.260185182094574 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9902861714363098 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9739024043083191 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6402878165245056 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.13596491515636444 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4238658845424652 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5827380418777466 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5487551689147949 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5533454418182373 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.6373456120491028 jet/accuracy_5_of_5‚0 *jet/accuracy_1_of_6‚0.9807692170143127 particle/accuracy_1_of_7‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.20631566643714905 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.46961477398872375 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3559505343437195 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7077948451042175 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.24008923768997192 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.012152778916060925 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.2103804498910904 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3030463755130768 +particle/specificity‚0.9523969888687134 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_5_of_6‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.8571428656578064 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9951314330101013 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2014327198266983 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.47666457295417786 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 :"Purity/*FRt2SRqqt2FBt/event_purity‚0.03999999910593033 jet/accuracy_4_of_7‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5402846932411194 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.15909092128276825 , Purity/3FRt0SRqqt1FBt/FRt_purity‚0.46875 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9918607473373413 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5384077429771423 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14888888597488403 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3541666567325592 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9869740009307861 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.17327585816383362 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5261628031730652 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 )jet/accuracy_2_of_4‚0.455057293176651 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1770833283662796 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.26906776428222656 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8205273747444153 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11554001271724701 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.026237770915031433 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 trainer/global_step‚12089 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20631779730319977 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.10673557966947556 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5813134908676147 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15348424017429352 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.32644665241241455 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5254629850387573 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.5107977986335754 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 8"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.503117024898529 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5887820720672607 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0023767082020640373 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5868169069290161 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,jet/accuracy_4_of_4‚0.008624290116131306 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4933333098888397 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.008620689623057842 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6451654434204102 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 )jet/accuracy_1_of_1‚0.511010468006134 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 particle/accuracy_2_of_4‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5573017597198486 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 particle/accuracy_1_of_6‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.48986920714378357 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9184397459030151 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.6021636128425598 7"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.73115074634552 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5567538738250732 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14702783524990082 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9912192225456238 +particle/sensitivity‚0.3487001657485962 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5468412041664124 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0024565381463617086 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 particle/accuracy_1_of_1‚1 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_1_of_7‚0.5 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2038288414478302 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9911760091781616 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5496430993080139 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5961805582046509 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_propo\Ü/N2rtion‚6.200397183420137e-05 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_3_of_7‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5347691774368286 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03750000149011612 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 particle/accuracy_5_of_7‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5855398178100586 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13475178182125092 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09233225882053375 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21113331615924835 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 )jet/accuracy_1_of_4‚0.926872968673706 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 )loss/val_total_loss‚40.60163180233506 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5592805743217468 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14757950603961945 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.13385175168514252 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5607690811157227 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.49000000953674316 jet/accuracy_5_of_7‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9979310631752014 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.47432440519332886 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0833333358168602 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9433051943778992 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9894494414329529 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5366320013999939 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.11934155970811844 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 jet/accuracy_5_of_6‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16896508634090424 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3897514045238495 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.14060531556606293 *jet/accuracy_1_of_2‚0.7765007019042969 jet/accuracy_2_of_7‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2748368978500366 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.5166666507720947 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.009290054440498352 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4861655533313751 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5532553195953369 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5608212351799011 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6312766671180725 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.1217186376452446 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.39027777314186096 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5948641300201416 particle/accuracy_7_of_7‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9845558404922485 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.23198391497135162 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5416667461395264 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9893516898155212 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.11932660639286041 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 ( Purity/3FRt1SRqqt2FBt/FRt_purity‚0.5 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17177154123783112 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3824692666530609 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20059846341609955 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9670635461807251 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5343391299247742 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02167479880154133 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 (particle/f_score‚0.43410661816596985 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5980483293533325 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1704304814338684 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.39460691809654236 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 % _timestamp‚1.7884782902040212e+09 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_5_of_5‚1 +jet/accuracy_3_of_6‚0.19230769574642181 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.16078153252601624 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_5‚1 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2666666507720947 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998404324054718 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5573981404304504 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.6681034564971924 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.4027777910232544 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.5000459551811218 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5800439119338989 particle/accuracy_2_of_7‚1 particle/accuracy_6_of_7‚1 _step‚993 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24109145998954773 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,jet/accuracy_3_of_3‚0.032434236258268356 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03995775431394577 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4444013237953186 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.525479257106781 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_7‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9916800856590271 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 9"Purity/0FRt2SRqqt2FBt/event_purity‚0.0714285746216774 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.15909288823604584 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.42592594027519226 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8991228342056274 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.019201921299099922 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9903996586799622 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6050086617469788 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6057794094085693 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.42094358801841736 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.04409223794937134 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_6_of_7‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 jet/accuracy_4_of_6‚0 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735632538795471 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5561522245407104 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2331567406654358 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13784068822860718 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.36203697323799133 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1Ax¨¤ÔM’0½§ & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.06711822748184204 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.18453028798103333 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.019607843831181526 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3046554625034332 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5947150588035583 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.041239891201257706 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2233007699251175 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4423363208770752 +jet/accuracy_3_of_4‚0.07943448424339294 _runtime‚14343.392206107 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.17178021371364594 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.30482909083366394 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5343220233917236 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 6 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.260185182094574 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9902861714363098 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9739024043083191 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6402878165245056 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.13596491515636444 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4238658845424652 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5827380418777466 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5487551689147949 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5533454418182373 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.6373456120491028 jet/accuracy_5_of_5‚0 *jet/accuracy_1_of_6‚0.9807692170143127 particle/accuracy_1_of_7‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.20631566643714905 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.46961477398872375 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3559505343437195 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7077948451042175 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.24008923768997192 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.012152778916060925 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.2103804498910904 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3030463755130768 +particle/specificity‚0.9523969888687134 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_5_of_6‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.8571428656578064 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9951314330101013 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2014327198266983 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.47666457295417786 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 :"Purity/*FRt2SRqqt2FBt/event_purity‚0.03999999910593033 jet/accuracy_4_of_7‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5402846932411194 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.15909092128276825 , Purity/3FRt0SRqqt1FBt/FRt_purity‚0.46875 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9918607473373413 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5384077429771423 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14888888597488403 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3541666567325592 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9869740009307861 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.17327585816383362 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5261628031730652 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 )jet/accuracy_2_of_4‚0.455057293176651 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1770833283662796 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.26906776428222656 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8205273747444153 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11554001271724701 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.026237770915031433 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 trainer/global_step‚12089 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20631779730319977 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.10673557966947556 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5813134908676147 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15348424017429352 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.32644665241241455 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5254629850387573 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.5107977986335754 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 8"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.503117024898529 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5887820720672607 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0023767082020640373 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5868169069290161 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,jet/accuracy_4_of_4‚0.008624290116131306 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4933333098888397 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.008620689623057842 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6451654434204102 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 )jet/accuracy_1_of_1‚0.511010468006134 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 particle/accuracy_2_of_4‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5573017597198486 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 particle/accuracy_1_of_6‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.48986920714378357 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9184397459030151 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.6021636128425598 7"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.73115074634552 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5567538738250732 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14702783524990082 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9912192225456238 +particle/sensitivity‚0.3487001657485962 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5468412041664124 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0024565381463617086 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 particle/accuracy_1_of_1‚1 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_1_of_7‚0.5 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2038288414478302 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9911760091781616 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5496430993080139 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5961805582046509 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_3_of_7‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5347691774368286 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03750000149011612 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 particle/accuracy_5_of_7‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5855398178100586 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13475178182125092 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09233225882053375 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21113331615924835 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 )jet/accuracy_1_of_4‚0.926872968673706 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 )loss/val_total_loss‚40.60163180233506 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5592805743217468 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14757950603961945 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.13385175168514252 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5607690811157227 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.49000000953674316 jet/accuracy_5_of_7‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9979310631752014 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.47432440519332886 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0833333358168602 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9433051943778992 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9894494414329529 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5366320013999939 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.11934155970811844 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 jet/accuracy_5_of_6‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16896508634090424 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3897514045238495 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.14060531556606293 *jet/accuracy_1_of_2‚0.7765007019042969 jet/accuracy_2_of_7‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2748368978500366 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.5166666507720947 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.009290054440498352 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4861655533313751 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5532553195953369 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5608212351799011 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6312766671180725 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.1217186376452446 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.39027777314186096 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5948641300201416 particle/accuracy_7_of_7‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9845558404922485 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.23198391497135162 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5416667461395264 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9893516898155212 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.11932660639286041 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 ( Purity/3FRt1SRqqt2FBt/FRt_purity‚0.5 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17177154123783112 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3824692666530609 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20059846341609955 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9670635461807251 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5343391299247742 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 @&Purity/0FRt1SRqqt3FBt/event_proporti·»x¤ðEon‚0.0004340277810115367 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02167479880154133 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 (particle/f_score‚0.43410661816596985 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5980483293533325 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1704304814338684 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.39460691809654236 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 % _timestamp‚1.7884782902040212e+09 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_5_of_5‚1 +jet/accuracy_3_of_6‚0.19230769574642181 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.16078153252601624 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_5‚1 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2666666507720947 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998404324054718 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5573981404304504 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.6681034564971924 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.4027777910232544 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.5000459551811218 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5800439119338989 particle/accuracy_2_of_7‚1 particle/accuracy_6_of_7‚1 _step‚993 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24109145998954773 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,jet/accuracy_3_of_3‚0.032434236258268356 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03995775431394577 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4444013237953186 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.525479257106781 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_7‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9916800856590271 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 9"Purity/0FRt2SRqqt2FBt/event_purity‚0.0714285746216774 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.15909288823604584 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.42592594027519226 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8991228342056274 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.019201921299099922 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9903996586799622 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6050086617469788 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6057794094085693 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.42094358801841736 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.04409223794937134 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_6_of_7‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 jet/accuracy_4_of_6‚0 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735632538795471 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5561522245407104 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2331567406654358 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13784068822860718 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.36203697323799133 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7651514410972595 particle/accuracy_2_of_2‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.564504086971283 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0011337868636474013 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.022392291575670242 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 <"Purity/2FRt1SRqqt1FBt/event_purity‚0.0059523810632526875 particle/accuracy_1_of_3‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17354938387870789 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9895833134651184 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9907757043838501 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5541612505912781 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1226053535938263 (particle/accuracy‚0.8581196069717407 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.026889055967330933 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5301781296730042 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23722390830516815 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.44767966866493225 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11926806718111038 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2023809552192688 +jet/accuracy_4_of_5‚0.02264465019106865 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5017858147621155 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.02380952425301075 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 particle/accuracy_4_of_7‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5564802289009094 +jet/accuracy_2_of_3‚0.27623873949050903 particle/accuracy_2_of_3‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.5057239532470703 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 epoch‚9 8"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.563230037689209 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5785521864891052 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5206280946731567 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.05349935591220856 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9322916865348816 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2938079237937927 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14555546641349792 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.38154762983322144 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5672968626022339 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1231442391872406 6 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.270114928483963 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0833333358168602 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11154208332300186 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.327725887298584 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_6‚1 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7348484396934509 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.019607843831181526 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5603302121162415 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12336037307977676 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2641419470310211 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4469697177410126 particle/accuracy_4_of_4‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983899593353271 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.12381001561880112 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974507093429565 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.5093567371368408 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.19949495792388916 particle/accuracy_3_of_4‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9910677671432495 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5357142686843872 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0015873016091063619 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9876037240028381 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.6987180113792419 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.992834210395813 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2014327198266983 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.5775985717773438 particle/accuracy_1_of_4‚1 6validation_average_jet_accuracy‚0.4586603045463562 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1846516728401184 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5468412041664124 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.28956836462020874 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.40640079975128174 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01189887523651123 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ) Purity/3FRt1SRqqt*FBt/FRt_purity‚0.25 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.16023391485214233 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.46666663885116577 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1626984179019928 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9917800426483154 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6681765913963318 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 8"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.584971010684967 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6360118985176086 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6491227746009827 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.4027777910232544 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15321384370326996 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.20802128314971924 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.33982861042022705 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.04784398153424263 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2014891654253006 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5534487366676331 *jet/accuracy_1_of_3‚0.8892883062362671áé½.õQ“0 booerk48j? Ö†èÔ¨÷ˆg0`Trainer.fit` stopped: `max_epochs=10` reached. r”0 booerk48j‹ ׆èÔ˜êWzEpoch 9/9 â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â” 1209/1209 0:22:36 • 0:00:00 0.91it/s v_num: rk48Má©P$•0 booerk48j ׆èԘˀX Ê6N<4–0/ _wandbruntime‚14350  _runtime‚14350†žCÔ—0‚(’Žp